Compare commits

..

2 Commits

Author SHA1 Message Date
Hauser Zhi e296adef35 Merge branch 'main' into update_flake_lock_action 2026-11-22 14:44:52 +08:00
houseme af0a20527d chore(deps): update flake.lock
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/2074cc6' (2026-11-12)
  → 'github:NixOS/nixpkgs/93a9ed0' (2026-11-21)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/5b87ee0' (2026-11-14)
  → 'github:oxalica/rust-overlay/52db205' (2026-11-21)
2026-11-22 05:07:43 +00:00
75 changed files with 9375 additions and 14202 deletions
-1
View File
@@ -57,7 +57,6 @@ rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
rustfs-sse 145 0
rustfs-storage-api 34 0
rustfs-targets 612 26
rustfs-test-utils 1 0
+1 -1
View File
@@ -231,7 +231,7 @@ jobs:
# --all-targets links the same test binaries as the reader lane.
CARGO_BUILD_JOBS: "2"
run: |
cargo build -p rustfs -p rustfs-ecstore -p rustfs-sse --all-targets --features rio-v2
cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
# Readers: the swift and sftp legs of test-and-lint-protocols. Built in
+2 -2
View File
@@ -563,7 +563,7 @@ jobs:
run: chmod go-w "$(realpath "$HOME")"
- name: Run rio-v2 clippy lints
run: cargo clippy -p rustfs -p rustfs-ecstore -p rustfs-sse --all-targets --features rio-v2 -- -D warnings
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
- name: Run rio-v2 feature tests
shell: bash
@@ -580,7 +580,7 @@ jobs:
rm -f target/nextest/ci/junit.xml
./scripts/ci/resource_sampler.sh start nextest-rio-v2
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore -p rustfs-sse --features rio-v2 \
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
2>&1 | tee artifacts/workspace/nextest.log
cargo test -p rustfs --doc --features rio-v2
+1 -1
View File
@@ -94,7 +94,7 @@ refactors.
|--------|--------------------------|----------------|
| Foundation | `checksums`, `common`, `config`, `data-usage`, `heal-contracts`, `scanner-metrics`, `utils` | Shared configuration, data-usage models, heal domain contracts, scanner telemetry types, utilities, and checksums. |
| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `s3-client`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, I/O pipelines, and the engine-side S3 client for remote tier/transition targets. |
| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `sse`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, server-side encryption key resolution, TLS, and security contracts. |
| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
-1
View File
@@ -116,7 +116,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
### Changed
- **Startup reports the state of `RUSTFS_SSE_S3_MASTER_KEY`**: the server now checks the local SSE-S3 master key when it starts and logs one `sse_local_master_key` event with `state` `loaded`, `not_set`, or `invalid`. An unusable value used to surface only as a `400 InvalidRequest` on the first SSE-S3 request served without a KMS; it is now logged as an error at startup with the reason. Startup still succeeds and request behavior is unchanged. The line "Using RUSTFS_SSE_S3_MASTER_KEY for SSE-S3 (KMS not configured)", logged when the key was first used, is replaced by this event.
- **Scanner bucket walkers are no longer tied to the disk count**: a new scanner key `walkers_per_disk` (`RUSTFS_SCANNER_WALKERS_PER_DISK`, `1` through `16`, default `1`) spawns that many bucket walkers per local disk of an erasure set, so a single-disk deployment can walk several buckets at once instead of one bucket, one object at a time. The walkers of one disk share its scanner admission, so a peer still sees one scan session per physical disk, and remote disks keep one walker. The walker count is what `max_concurrent_disk_scans` caps and the disk count stays the floor; the `disk_concurrency_budget` log line now reports `walkers_per_disk`, `walkers`, and the `reason` for the effective limit. The default keeps today's I/O profile because a local disk cannot yet report whether it is rotational; the design intends a higher default on flash once it can (`docs/architecture/scanner-per-bucket-convergence.md`). As groundwork for pacing, the raw-enumeration page writer inserts into its building page in place and commits pages without copying them, and a regression test pins the cost of recording 20,000 entries in one prefix so the quadratic bookkeeping removed by [#8114](https://github.com/rustfs/rustfs/pull/8114) (rustfs#8113) cannot return. No persisted format, protocol or admin API changes beyond the additional config key, which older releases preserve and ignore.
- **Encryption and KMS work merged since `1.0.0-rc.5`** (entries were missing from this section):
- **Persisted KMS configuration secrets** are sealed field-by-field with `RUSTFS_KMS_CONFIG_SECRET`. **When the variable is unset the secrets are persisted in cleartext and the server only warns** (`persisted KMS configuration carries cleartext secrets`); it never refuses the write. Set it, identically, on every node, and re-save the configuration to seal an existing one.
Generated
-28
View File
@@ -9959,7 +9959,6 @@ dependencies = [
"rustfs-scanner-metrics",
"rustfs-security-governance",
"rustfs-signer",
"rustfs-sse",
"rustfs-storage-api",
"rustfs-targets",
"rustfs-test-utils",
@@ -11429,33 +11428,6 @@ dependencies = [
"url",
]
[[package]]
name = "rustfs-sse"
version = "1.0.5"
dependencies = [
"aes-gcm",
"async-trait",
"axum",
"base64-simd",
"chacha20poly1305",
"hmac 0.13.0",
"http 1.5.0",
"md-5",
"rand 0.10.4",
"rustfs-config",
"rustfs-kms",
"rustfs-policy",
"rustfs-s3-types",
"rustfs-utils",
"serde",
"serde_json",
"sha2 0.11.1",
"temp-env",
"tempfile",
"tokio",
"tracing",
]
[[package]]
name = "rustfs-storage-api"
version = "1.0.5"
-2
View File
@@ -60,7 +60,6 @@ members = [
"crates/security-governance", # Security governance contracts
"crates/extension-schema", # Extension schema contracts
"crates/signer", # client signer
"crates/sse", # S3 server-side encryption key resolution and metadata
"crates/storage-api", # Storage API contracts
"crates/test-utils", # Shared test bootstrap helpers (dev-dependency only)
"crates/targets", # Target-specific configurations and utilities
@@ -141,7 +140,6 @@ rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-sse = { path = "crates/sse", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
+1 -2
View File
@@ -6,8 +6,7 @@ Applies to `crates/kms/`.
When changing key-management behavior, verify compatibility with:
- `crates/sse/` (SSE key resolution; consumes this crate on the data path)
- `rustfs/src/storage/sse.rs` (binds `crates/sse/` to the running server)
- `rustfs/src/storage/sse.rs`
- `rustfs/src/admin/handlers/kms.rs`
- `rustfs/src/admin/handlers/kms_dynamic.rs`
- `rustfs/src/admin/handlers/kms_keys.rs`
-57
View File
@@ -1,57 +0,0 @@
# SSE Crate Instructions
Applies to `crates/sse/`.
## Boundary
- This crate sits between the S3 layer and the storage engine and depends on neither. Its `rustfs-*` dependencies are pinned by `scripts/check_architecture_migration_rules.sh` to `rustfs-config`, `rustfs-kms`, `rustfs-policy`, `rustfs-s3-types`, and `rustfs-utils`; see [crate boundaries](../../docs/architecture/crate-boundaries.md).
- Anything the crate needs from the running server arrives through `SseHost` (`src/host.rs`), passed as the first argument of every entry point that consults the server. Do not add a global that stands in for the host: a caller must not be able to reach key selection without naming the server it speaks for.
- Translation to and from storage-engine types (`ObjectEncryptionResolver`, `WriteEncryption`, `MultipartCompletionAuthorization`) and request types belongs in `rustfs/src/storage/sse.rs`, not here.
- Report failures as `SseError`, classified with the S3 error code the caller answers with. A failure that did not originate from the request goes through `SseError::internal`, which keeps its detail out of the response.
## Layout
- `src/startup.rs` settles the state that is fixed for the lifetime of the process and reports it. Nothing that follows the KMS service state belongs there: the service follows configuration and can be started, restarted, or stopped from the admin API, so every choice that depends on it is made per request.
- `src/core/` holds the entry points (`write.rs`, `read.rs`, `rewrap.rs`) and the request principal (`request.rs`). An entry point only orders the steps; the decisions live in the two layers below it.
- `src/core/s3/` is what S3 defines: the SSE headers of a request and a response, the scheme a write or a stored object resolves to, the SSE-C checks, and the authorization of a KMS key.
- `src/core/keying/` is how RustFS forms, seals, and records an object's key. `direct.rs` and `object.rs` are the two modes, and each owns both directions of its stored record: the code that writes a key's metadata sits next to the code that reads it back. `mod.rs` is the only place that says which mode a build writes; `stored.rs` holds the order in which a read recognizes the modes and picks the provider that unwraps the data key; `metadata.rs` records the scheme and lists every stored key name.
- `src/kms/` holds the data-key providers: they wrap and unwrap a data key and decide nothing about schemes. The write and rewrap entry points, `core/keying/stored.rs`, and the startup check of the local master key call them; `core/s3/` never does.
Rules that keep the layers apart:
- `core/keying/` never imports from `core/s3/`. `core/s3/` reads stored metadata only through `core/keying/metadata.rs`. Anything else that needs both layers is joined in an entry point.
- `#[cfg(feature = "rio-v2")]` appears only under `core/keying/` and `kms/`. An entry point or `core/s3/` that needs a build-dependent answer asks `core/keying/` for it.
- A new stored format, including a MinIO-compatible one, is added under `core/keying/` (and `kms/` when it brings its own wrapping) without changes under `core/s3/`: a mode file with its record and read functions, one entry in the recognition order of `stored.rs`, and its key names in the lists of `metadata.rs`.
- A read decides from what the object recorded. The scheme a request advertises and whether a KMS is running now are write policy and runtime state; neither says how an existing object was keyed.
## Security
- Never log plaintext customer keys, decrypted key material, data-key envelopes, nonces, or SSE request payloads, including through error strings and `Debug` output.
- KMS audit tags may carry only the fields documented on `KmsRequestAuditState`.
- The `test-util` feature exposes authorization overrides and DEK-provider injection. It is a dev-dependency feature only; never enable it from a normal dependency.
## Change Coordination
When changing key resolution, metadata, or error classification, verify compatibility with:
- `rustfs/src/storage/sse.rs` (the host implementation and the engine-boundary adapters)
- `crates/kms/`
- `crates/utils/src/http/object_encryption_keys.rs` (the stored-metadata key vocabulary)
- `rustfs/src/kms_rekey.rs` and `rustfs/src/admin/handlers/kms*.rs`
## Testing
Run both builds; the `rio-v2` feature swaps the object-key scheme and compiles different code paths:
```bash
cargo nextest run -p rustfs-sse --no-tests fail
cargo nextest run -p rustfs-sse --features rio-v2 --no-tests fail
```
The adapter tests live with the adapter:
```bash
cargo nextest run -p rustfs --no-tests fail -E 'test(/^storage::sse::/)'
```
A change that should leave client-visible behavior or the stored form untouched (a refactor, a move between layers) is checked against a real server: run `scripts/test/sse_blackbox/` on the binary before and after the change, compare the two runs, and cross-read their data directories. Its README has the commands.
-70
View File
@@ -1,70 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[package]
name = "rustfs-sse"
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
homepage.workspace = true
description = "S3 server-side encryption (SSE-S3, SSE-KMS, SSE-C) key resolution and metadata for RustFS."
keywords = ["sse", "encryption", "s3", "kms", "rustfs"]
categories = ["cryptography"]
[lints]
workspace = true
[features]
default = []
# MinIO-compatible sealed object keys (DARE). Mirrors the `rio-v2` feature of
# the server binary, which enables both together.
rio-v2 = ["dep:chacha20poly1305", "dep:hmac", "dep:sha2"]
# Test seams for dependents: per-principal authorization overrides, DEK
# provider injection and metadata fixtures. Never enable in a release build.
test-util = []
[dependencies]
# The only RustFS crates this one may reach. It sits between the S3 layer and
# the storage engine and must not depend on either: what it needs from the
# running server arrives through `SseHost`.
rustfs-config = { workspace = true }
rustfs-kms = { workspace = true }
rustfs-policy = { workspace = true }
rustfs-s3-types = { workspace = true }
rustfs-utils = { workspace = true, features = ["http", "path"] }
aes-gcm = { workspace = true, features = ["rand_core", "zeroize"] }
async-trait = { workspace = true }
base64-simd = { workspace = true }
chacha20poly1305 = { workspace = true, optional = true }
hmac = { workspace = true, optional = true }
http = { workspace = true }
md-5 = { workspace = true }
rand = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
sha2 = { workspace = true, optional = true }
tracing = { workspace = true }
[dev-dependencies]
# The fake AWS KMS endpoint the backend-selection tests talk to.
axum = { workspace = true }
temp-env = { workspace = true, features = ["async_closure"] }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["macros", "net", "rt-multi-thread", "sync", "time"] }
[lib]
doctest = false
-292
View File
@@ -1,292 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-request summary of data-plane KMS use, attached to the request's audit entry.
use crate::core::request::SseKmsPrincipal;
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_kms::is_data_key_envelope;
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::http::object_encryption_keys::{INTERNAL_ENCRYPTION_KEY_HEADER, SSEType, normalize_managed_metadata};
use serde_json::Value;
use std::collections::{BTreeSet, HashMap};
use std::sync::{Arc, LazyLock, Mutex, Weak};
/// Tag keys carrying the KMS summary on an S3 audit entry.
///
/// Audit consumers key off these strings, so they are a wire contract: append
/// new keys rather than renaming existing ones.
const KMS_AUDIT_TAG_SSE_TYPE: &str = "sseType";
const KMS_AUDIT_TAG_KEY_ID: &str = "kmsKeyId";
const KMS_AUDIT_TAG_KEY_VERSION: &str = "kmsKeyVersion";
const KMS_AUDIT_TAG_OUTCOME: &str = "kmsOutcome";
const KMS_AUDIT_TAG_ERROR_CLASS: &str = "kmsErrorClass";
/// Separator for the rare request that touched more than one key or scheme.
const KMS_AUDIT_VALUE_SEPARATOR: &str = ",";
/// What the data path did with KMS while serving one S3 request.
///
/// This rides on the request's existing S3 audit entry instead of becoming its
/// own event: the entry already carries the principal, request ID and API, so
/// attaching the KMS outcome costs no extra event volume and needs no second
/// correlation key.
///
/// # Redaction
///
/// Only the fields below may ever be recorded. The S3 audit entry fans out to
/// every configured target, several of which sit outside the KMS trust
/// boundary, so nothing derived from a data key — plaintext, ciphertext,
/// envelope, nonce — nor any caller-supplied encryption-context value belongs
/// here. Key identifiers and scheme names are configuration, not secrets.
#[derive(Debug, Default)]
struct KmsRequestAuditState {
sse_types: BTreeSet<&'static str>,
key_ids: BTreeSet<String>,
key_versions: BTreeSet<u32>,
/// Set by the first failure. A request that failed any KMS interaction is
/// reported as failed even if others succeeded: a partially completed
/// envelope operation is not a success for an audit reader.
error_class: Option<&'static str>,
recorded: bool,
}
/// Shared accumulator for one request's KMS summary.
///
/// Written by the managed-SSE code paths through the request's principal, read
/// once through [`KmsRequestAuditScope::audit_tags`] by whoever owns the
/// request's audit entry.
#[derive(Debug, Default)]
pub(crate) struct KmsRequestAudit(Mutex<KmsRequestAuditState>);
impl KmsRequestAudit {
pub(crate) fn record(
&self,
sse_type: SSEType,
key_id: Option<&str>,
key_version: Option<u32>,
error_class: Option<&'static str>,
) {
// A poisoned lock costs the audit entry its KMS tags; it must never cost
// the request its result.
let Ok(mut state) = self.0.lock() else {
return;
};
state.recorded = true;
state.sse_types.insert(sse_type.audit_label());
if let Some(key_id) = key_id.filter(|value| !value.is_empty()) {
state.key_ids.insert(key_id.to_string());
}
if let Some(key_version) = key_version {
state.key_versions.insert(key_version);
}
if state.error_class.is_none() {
state.error_class = error_class;
}
}
/// Render the summary as audit tags, or nothing when the request performed
/// no KMS work at all (SSE-C and unencrypted objects never reach KMS).
pub fn audit_tags(&self) -> Vec<(&'static str, Value)> {
let Ok(state) = self.0.lock() else {
return Vec::new();
};
if !state.recorded {
return Vec::new();
}
let mut tags = Vec::with_capacity(5);
tags.push((KMS_AUDIT_TAG_SSE_TYPE, join_audit_values(state.sse_types.iter().copied())));
if !state.key_ids.is_empty() {
tags.push((KMS_AUDIT_TAG_KEY_ID, join_audit_values(state.key_ids.iter().map(String::as_str))));
}
if !state.key_versions.is_empty() {
tags.push((
KMS_AUDIT_TAG_KEY_VERSION,
join_audit_values(state.key_versions.iter().map(u32::to_string)),
));
}
let outcome = if state.error_class.is_some() { "failure" } else { "success" };
tags.push((KMS_AUDIT_TAG_OUTCOME, Value::String(outcome.to_string())));
if let Some(error_class) = state.error_class {
tags.push((KMS_AUDIT_TAG_ERROR_CLASS, Value::String(error_class.to_string())));
}
tags
}
}
/// Render a set of values as one tag value.
///
/// Always a string, including for the single-value case: a consumer that would
/// have to branch on the JSON type of a tag is a consumer that will get it
/// wrong for the request that happens to touch two keys.
fn join_audit_values(values: impl Iterator<Item = impl AsRef<str>>) -> Value {
let joined = values.map(|value| value.as_ref().to_string()).collect::<Vec<_>>();
Value::String(joined.join(KMS_AUDIT_VALUE_SEPARATOR))
}
/// Slots of the requests currently being audited, keyed by canonical request ID.
///
/// Only a weak reference is held: the owning [`KmsRequestAuditScope`] decides the
/// lifetime, so a scope that is somehow not dropped cannot keep a slot alive.
static KMS_REQUEST_AUDITS: LazyLock<Mutex<HashMap<String, Weak<KmsRequestAudit>>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
/// Registration of one request's KMS audit slot, dropped with the request's
/// audit entry.
pub struct KmsRequestAuditScope {
request_id: String,
audit: Arc<KmsRequestAudit>,
}
impl KmsRequestAuditScope {
/// Open a slot for `request_id`, so managed-SSE operations served under it can
/// be summarised onto its audit entry.
pub fn register(request_id: &str) -> Self {
let audit = Arc::new(KmsRequestAudit::default());
if let Ok(mut registry) = KMS_REQUEST_AUDITS.lock() {
registry.insert(request_id.to_string(), Arc::downgrade(&audit));
}
Self {
request_id: request_id.to_string(),
audit,
}
}
/// Tags describing the KMS work recorded under this request.
pub fn audit_tags(&self) -> Vec<(&'static str, Value)> {
self.audit.audit_tags()
}
}
impl Drop for KmsRequestAuditScope {
fn drop(&mut self) {
if let Ok(mut registry) = KMS_REQUEST_AUDITS.lock() {
registry.remove(&self.request_id);
}
}
}
/// Resolve the audit slot opened for `request_id`, if the request is being audited.
pub(crate) fn kms_request_audit(request_id: &str) -> Option<Arc<KmsRequestAudit>> {
KMS_REQUEST_AUDITS.lock().ok()?.get(request_id)?.upgrade()
}
/// Failure classes for data-path failures that never reached a KMS backend.
///
/// Backend failures carry the classification defined by the KMS audit contract;
/// everything else is classified at the S3 boundary it surfaced through.
pub(crate) fn kms_data_plane_error_class(error: &SseError) -> &'static str {
if let Some(failure) = error
.source
.as_ref()
.and_then(|source| source.downcast_ref::<KmsDataPlaneFailure>())
{
return failure.class;
}
match error.code {
S3ErrorCode::AccessDenied => "access_denied",
S3ErrorCode::InvalidArgument | S3ErrorCode::InvalidRequest => "invalid_argument",
_ => "sse_internal",
}
}
/// Carries a KMS failure class across the conversion to an S3-facing error.
///
/// The class is decided while the `KmsError` is still typed; re-deriving it from
/// a rendered message downstream would be guesswork.
#[derive(Debug)]
pub(crate) struct KmsDataPlaneFailure {
pub(crate) class: &'static str,
pub(crate) source: Option<Box<dyn std::error::Error + Send + Sync>>,
}
impl std::fmt::Display for KmsDataPlaneFailure {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match &self.source {
Some(source) => source.fmt(formatter),
None => formatter.write_str(self.class),
}
}
}
impl std::error::Error for KmsDataPlaneFailure {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
self.source
.as_ref()
.map(|source| source.as_ref() as &(dyn std::error::Error + 'static))
}
}
/// Record the outcome of one managed-SSE operation on the request's audit entry.
///
/// A `None` principal marks an internal caller — replication, lifecycle, heal —
/// which has no S3 audit entry to attach to. `key_version` is a closure for
/// exactly that caller: extracting the version means base64-decoding and
/// parsing the stored envelope, work that must not run on the internal hot
/// paths that discard it.
pub(crate) fn record_managed_kms_outcome(
principal: Option<&SseKmsPrincipal>,
sse_type: SSEType,
key_id: Option<&str>,
key_version: impl FnOnce() -> Option<u32>,
result: Result<(), &SseError>,
) {
let Some(audit) = principal.and_then(|principal| principal.request_audit.as_ref()) else {
return;
};
audit.record(sse_type, key_id, key_version(), result.err().map(kms_data_plane_error_class));
}
/// Master-key version recorded in a managed-SSE data-key envelope, if the
/// wrapping backend recorded one.
///
/// `None` is the honest answer for every other shape: Transit and AWS keep
/// the version inside their own opaque ciphertext, Local records no version,
/// and pre-versioning envelopes never carried the field. The single field is
/// read through `serde_json::Value` rather than a full `DataKeyEnvelope`
/// parse, so the audit path cannot double-count the envelope's unknown-field
/// observability and touches nothing else in the envelope.
pub(crate) fn envelope_master_key_version(envelope_bytes: &[u8]) -> Option<u32> {
if !is_data_key_envelope(envelope_bytes) {
return None;
}
u32::try_from(
serde_json::from_slice::<Value>(envelope_bytes)
.ok()?
.get("master_key_version")?
.as_u64()?,
)
.ok()
}
/// Master-key version of the envelope stored on an object, for the audit
/// summary of a read against that object.
pub(crate) fn stored_envelope_master_key_version(metadata: &HashMap<String, String>) -> Option<u32> {
// No context recoder: the recode only ever inserts the context key, which
// this lookup never reads, so the normalized result is identical without it.
let encoded = normalize_managed_metadata(metadata, None);
let encoded = encoded.get(INTERNAL_ENCRYPTION_KEY_HEADER)?;
let envelope = BASE64_STANDARD.decode_to_vec(encoded).ok()?;
envelope_master_key_version(&envelope)
}
-67
View File
@@ -1,67 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The encryption context a data key is bound to.
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_kms::types::ObjectEncryptionContext;
use rustfs_utils::http::object_encryption_keys::MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER;
use rustfs_utils::path::path_join_buf;
use std::collections::HashMap;
pub(crate) fn canonical_kms_bucket_path(bucket: &str, key: &str) -> String {
path_join_buf(&[bucket, key])
}
pub(crate) fn build_kms_request_context(
bucket: &str,
key: &str,
provided_context: Option<&HashMap<String, String>>,
) -> HashMap<String, String> {
let mut context = provided_context.cloned().unwrap_or_default();
context
.entry(bucket.to_string())
.or_insert_with(|| canonical_kms_bucket_path(bucket, key));
context
}
pub(crate) fn build_object_encryption_context(
bucket: &str,
key: &str,
provided_context: Option<&HashMap<String, String>>,
) -> ObjectEncryptionContext {
let mut context = ObjectEncryptionContext::new(bucket.to_string(), key.to_string());
for (ctx_key, ctx_value) in build_kms_request_context(bucket, key, provided_context) {
context = context.with_encryption_context(ctx_key, ctx_value);
}
context
}
pub(crate) fn encode_minio_kms_context(context: &HashMap<String, String>) -> Result<String, SseError> {
let encoded = serde_json::to_vec(context).map_err(|e| SseError::internal(format!("Failed to serialize KMS context: {e}")))?;
Ok(BASE64_STANDARD.encode_to_string(encoded))
}
pub(crate) fn decode_minio_kms_context(metadata: &HashMap<String, String>) -> Result<Option<HashMap<String, String>>, SseError> {
let Some(encoded) = metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER) else {
return Ok(None);
};
let decoded = BASE64_STANDARD
.decode_to_vec(encoded)
.map_err(|e| SseError::internal(format!("Failed to decode MinIO KMS context: {e}")))?;
serde_json::from_slice(&decoded)
.map(Some)
.map_err(|e| SseError::internal(format!("Failed to parse MinIO KMS context: {e}")))
}
-212
View File
@@ -1,212 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Direct keying: the data key encrypts the object itself, under a stored base nonce.
#[cfg(not(feature = "rio-v2"))]
use crate::core::keying::NewObjectKey;
#[cfg(not(feature = "rio-v2"))]
use crate::core::keying::material::EncryptionKeyKind;
use crate::core::keying::{StoredDataKey, StoredKeyMode};
use crate::error::SseError;
use crate::host::SseHost;
use base64_simd::STANDARD as BASE64_STANDARD;
use md5::{Digest as Md5Digest, Md5};
#[cfg(not(feature = "rio-v2"))]
use rand::Rng;
#[cfg(not(feature = "rio-v2"))]
use rustfs_kms::DataKey;
#[cfg(not(feature = "rio-v2"))]
use rustfs_utils::http::object_encryption_keys::SSEType;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_IV_HEADER, INTERNAL_ENCRYPTION_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
};
use std::collections::HashMap;
/// SSE-C: record the base nonce the object is encrypted under.
///
/// The nonce is random, so decrypt reads it back instead of recomputing a
/// deterministic value. It is written under both the RustFS and the MinIO key per
/// repo convention. This covers a single PUT and a multipart session alike:
/// CreateMultipartUpload builds the session metadata through the same call.
pub(crate) fn record_customer_key_nonce(metadata: &mut HashMap<String, String>, base_nonce: [u8; 12]) {
metadata.insert(INTERNAL_ENCRYPTION_IV_HEADER.to_string(), BASE64_STANDARD.encode_to_string(base_nonce));
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(base_nonce),
);
}
/// Managed SSE: record the wrapped data key, the nonce issued with it, and the cipher.
pub(crate) fn record_wrapped_data_key(
metadata: &mut HashMap<String, String>,
wrapped_data_key: &[u8],
base_nonce: [u8; 12],
algorithm: &str,
) {
metadata.insert(
INTERNAL_ENCRYPTION_KEY_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(wrapped_data_key),
);
metadata.insert(INTERNAL_ENCRYPTION_IV_HEADER.to_string(), BASE64_STANDARD.encode_to_string(base_nonce));
metadata.insert(INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), algorithm.to_string());
}
/// Managed SSE: read back what [`record_wrapped_data_key`] stored.
///
/// `normalized_metadata` is the object's metadata with the managed keys normalized.
pub(crate) async fn read_wrapped_data_key(
host: &dyn SseHost,
normalized_metadata: &HashMap<String, String>,
) -> Result<StoredDataKey, SseError> {
let (wrapped, iv, algorithm) = if let Some(service) = host.encryption_service().await {
// Production mode: use service for metadata parsing
let parsed = service
.headers_to_metadata(normalized_metadata)
.map_err(|e| SseError::internal(format!("Failed to parse encryption metadata: {e}")))?;
(parsed.encrypted_data_key, parsed.iv, parsed.algorithm)
} else {
// Test mode: parse metadata manually
let encrypted_key_b64 = normalized_metadata
.get(INTERNAL_ENCRYPTION_KEY_HEADER)
.ok_or_else(|| SseError::internal("Missing encrypted key in metadata"))?;
let encrypted_data_key = BASE64_STANDARD
.decode_to_vec(encrypted_key_b64)
.map_err(|e| SseError::internal(format!("Failed to decode encrypted key: {e}")))?;
let iv_b64 = normalized_metadata
.get(INTERNAL_ENCRYPTION_IV_HEADER)
.ok_or_else(|| SseError::internal("Missing IV in metadata"))?;
let iv = BASE64_STANDARD
.decode_to_vec(iv_b64)
.map_err(|e| SseError::internal(format!("Failed to decode IV: {e}")))?;
let algorithm = normalized_metadata
.get(INTERNAL_ENCRYPTION_ALGORITHM_HEADER)
.cloned()
.unwrap_or_else(|| "AES256".to_string());
(encrypted_data_key, iv, algorithm)
};
let base_nonce = <[u8; 12]>::try_from(iv.as_slice())
.map_err(|_| SseError::internal("Invalid encryption nonce length; expected 12 bytes"))?;
Ok(StoredDataKey {
wrapped,
algorithm,
mode: StoredKeyMode::Direct { base_nonce },
})
}
/// Resolve the SSE-C Direct base nonce for decryption.
///
/// New objects persist a random nonce at encryption time (see
/// [`record_customer_key_nonce`]) under `INTERNAL_ENCRYPTION_IV_HEADER` (preferred) or,
/// for MinIO interop, `MINIO_INTERNAL_ENCRYPTION_IV_HEADER`. Legacy objects written before
/// random nonces were persisted carry no stored IV; for those we fall back to the
/// deterministic `generate_ssec_nonce(bucket, key)` value they were originally encrypted
/// with, so previously stored objects still decrypt correctly.
pub(crate) fn read_stored_ssec_nonce(metadata: &HashMap<String, String>, bucket: &str, key: &str) -> [u8; 12] {
metadata
.get(INTERNAL_ENCRYPTION_IV_HEADER)
.or_else(|| metadata.get(MINIO_INTERNAL_ENCRYPTION_IV_HEADER))
.and_then(|encoded| BASE64_STANDARD.decode_to_vec(encoded).ok())
.and_then(|bytes| <[u8; 12]>::try_from(bytes.as_slice()).ok())
.unwrap_or_else(|| generate_ssec_nonce(bucket, key))
}
/// Generate deterministic nonce for SSE-C encryption
///
/// The nonce is derived from the bucket and key to ensure:
/// 1. Same object always gets the same nonce (required for SSE-C)
/// 2. Different objects get different nonces
pub(crate) fn generate_ssec_nonce(bucket: &str, key: &str) -> [u8; 12] {
let nonce_source = format!("{bucket}-{key}");
let nonce_hash: [u8; 16] = Md5::digest(nonce_source.as_bytes()).into();
let mut nonce = [0u8; 12];
nonce.copy_from_slice(&nonce_hash[..12]);
nonce
}
/// Whether a multipart session needs the customer key when it is created.
///
/// Direct mode records only a nonce for the session; the customer key is checked
/// per part, against what the session recorded.
#[cfg(not(feature = "rio-v2"))]
pub(crate) const SESSION_KEY_NEEDS_CUSTOMER_KEY: bool = false;
#[cfg(not(feature = "rio-v2"))]
fn random_base_nonce() -> [u8; 12] {
let mut base_nonce = [0u8; 12];
rand::rng().fill_bytes(&mut base_nonce);
base_nonce
}
/// SSE-C object: the customer key encrypts the object itself.
///
/// Uses a fresh random nonce per encryption. A deterministic nonce derived from
/// (bucket, key) would repeat whenever the same object is overwritten under the same
/// SSE-C key, reusing an identical (key, nonce) pair and catastrophically breaking
/// AES-256-GCM. The nonce is persisted (see [`record_customer_key_nonce`]) and read
/// back on decrypt (see [`read_stored_ssec_nonce`]).
#[cfg(not(feature = "rio-v2"))]
pub(crate) fn new_key_under_customer_key(customer_key: [u8; 32], _bucket: &str, _key: &str) -> Result<NewObjectKey, SseError> {
Ok(NewObjectKey {
key_bytes: customer_key,
base_nonce: random_base_nonce(),
key_kind: EncryptionKeyKind::Direct,
sealed_key: None,
})
}
/// SSE-C multipart session: a nonce every part of the upload shares.
///
/// Generates a real random nonce for the session and persists it (see
/// [`record_customer_key_nonce`]). Every part of this upload reads the same
/// persisted nonce back, so all parts share one nonce without reintroducing the
/// deterministic (bucket, key) reuse hazard. `key_bytes` stays a placeholder: it is
/// neither used for encryption nor persisted; the real customer key is validated per
/// part on upload.
#[cfg(not(feature = "rio-v2"))]
pub(crate) fn new_session_key_under_customer_key(
_customer_key: Option<[u8; 32]>,
_bucket: &str,
_key: &str,
) -> Result<NewObjectKey, SseError> {
Ok(NewObjectKey {
key_bytes: [0; 32],
base_nonce: random_base_nonce(),
key_kind: EncryptionKeyKind::Direct,
sealed_key: None,
})
}
/// Managed SSE: the data key encrypts the object itself, under the nonce issued with it.
#[cfg(not(feature = "rio-v2"))]
pub(crate) fn new_key_under_data_key(
data_key: &DataKey,
_sse_type: SSEType,
_bucket: &str,
_key: &str,
) -> Result<NewObjectKey, SseError> {
Ok(NewObjectKey {
key_bytes: data_key.plaintext_key,
base_nonce: data_key.nonce,
key_kind: EncryptionKeyKind::Direct,
sealed_key: None,
})
}
-115
View File
@@ -1,115 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The key material an encrypted write or read runs on.
#[cfg(feature = "rio-v2")]
use crate::core::keying::object::{SEALED_KEY_IV_SIZE, SEALED_KEY_SIZE};
use rustfs_utils::http::object_encryption_keys::SSEType;
use std::collections::HashMap;
/// Encryption material returned by [`sse_encryption`] / [`sse_prepare_encryption`].
pub struct EncryptionMaterial {
#[allow(unused)]
pub sse_type: SSEType,
pub server_side_encryption: String,
pub kms_key_id: Option<String>,
#[allow(unused)]
pub algorithm: String,
/// Encryption key bytes
pub key_bytes: [u8; 32],
/// Base nonce/IV used by rio to derive block/part nonces.
pub base_nonce: [u8; 12],
/// Encrypted DEK for managed SSE. Absent for SSE-C.
pub encrypted_data_key: Option<Vec<u8>>,
/// SSE-C key MD5 if customer-managed encryption is in use.
pub customer_key_md5: Option<String>,
/// Original plaintext size when it should be persisted alongside metadata.
pub original_size: Option<i64>,
/// Indicates whether `key_bytes` is a direct stream key or a MinIO-style object key.
pub key_kind: EncryptionKeyKind,
/// Original client-provided SSE-KMS context, stored in MinIO-compatible metadata.
pub managed_kms_context: Option<HashMap<String, String>>,
/// MinIO-compatible sealed object-key metadata for managed SSE object-key mode.
#[cfg_attr(not(feature = "rio-v2"), allow(dead_code))]
pub managed_sealed_key: Option<ManagedSealedKey>,
}
/// Decryption material returned by [`sse_decryption`].
pub struct DecryptionMaterial {
#[allow(unused)]
pub sse_type: SSEType,
pub server_side_encryption: String,
pub kms_key_id: Option<String>,
#[allow(unused)]
pub algorithm: String,
pub customer_key_md5: Option<String>, // if use SSE-C, check key md5
/// Decryption key bytes
pub key_bytes: [u8; 32],
/// Base nonce/IV used by rio to derive block/part nonces.
pub base_nonce: [u8; 12],
/// Indicates whether `key_bytes` is a direct stream key or a MinIO-style object key.
pub key_kind: EncryptionKeyKind,
}
impl std::fmt::Debug for EncryptionMaterial {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("EncryptionMaterial")
.field("sse_type", &self.sse_type)
.field("server_side_encryption", &self.server_side_encryption)
.field("original_size", &self.original_size)
.finish_non_exhaustive()
}
}
impl std::fmt::Debug for DecryptionMaterial {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("DecryptionMaterial")
.field("sse_type", &self.sse_type)
.field("server_side_encryption", &self.server_side_encryption)
.finish_non_exhaustive()
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EncryptionKeyKind {
Direct,
Object,
}
#[derive(Debug, Clone)]
pub struct ManagedSealedKey {
#[cfg(feature = "rio-v2")]
pub iv: [u8; SEALED_KEY_IV_SIZE],
#[cfg(feature = "rio-v2")]
pub sealed_key: [u8; SEALED_KEY_SIZE],
}
impl EncryptionMaterial {
/// The KMS key id a write response may advertise.
///
/// `kms_key_id` is always set for managed SSE because SSE-S3 also wraps
/// its data key under the service default key, but that key is an
/// internal detail of an `AES256` object: only an `aws:kms` object names
/// a key the caller can act on, and `x-amz-server-side-encryption-aws-kms-key-id`
/// is defined only for that scheme.
pub fn response_kms_key_id(&self) -> Option<String> {
matches!(self.sse_type, SSEType::SseKms)
.then(|| self.kms_key_id.clone())
.flatten()
}
}
-270
View File
@@ -1,270 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Encryption metadata as it is stored with the object.
use crate::core::keying::context::encode_minio_kms_context;
use crate::core::keying::direct::{record_customer_key_nonce, record_wrapped_data_key};
use crate::core::keying::material::{EncryptionKeyKind, EncryptionMaterial};
#[cfg(feature = "rio-v2")]
use crate::core::keying::object::{record_data_key_in_minio_slots, record_sealed_customer_key};
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_CONTEXT_HEADER, INTERNAL_ENCRYPTION_IV_HEADER,
INTERNAL_ENCRYPTION_KEY_HEADER, INTERNAL_ENCRYPTION_KEY_ID_HEADER, INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER,
INTERNAL_ENCRYPTION_TAG_HEADER, MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER, MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER, MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER, SSEC_ORIGINAL_SIZE_HEADER, SSEType,
};
use std::borrow::Cow;
use std::collections::HashMap;
/// Two spellings of one encryption metadata key carry different values.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ConflictingEncryptionMetadata {
/// The canonical key whose case-variants disagree.
pub key: &'static str,
}
impl std::fmt::Display for ConflictingEncryptionMetadata {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(formatter, "conflicting object encryption metadata for {}", self.key)
}
}
impl std::error::Error for ConflictingEncryptionMetadata {}
/// Fold case-variant spellings of the encryption metadata keys onto their canonical names.
///
/// # Errors
///
/// When two spellings of the same key hold different values: picking either
/// would decrypt with material the object may not have been written under.
pub fn normalize_encryption_metadata_case(
metadata: &HashMap<String, String>,
) -> Result<Cow<'_, HashMap<String, String>>, ConflictingEncryptionMetadata> {
const CANONICAL_KEYS: &[&str] = &[
"x-amz-server-side-encryption",
"x-amz-server-side-encryption-aws-kms-key-id",
"x-amz-server-side-encryption-customer-algorithm",
"x-amz-server-side-encryption-customer-key-md5",
SSEC_ORIGINAL_SIZE_HEADER,
INTERNAL_ENCRYPTION_KEY_ID_HEADER,
INTERNAL_ENCRYPTION_KEY_HEADER,
INTERNAL_ENCRYPTION_ALGORITHM_HEADER,
INTERNAL_ENCRYPTION_IV_HEADER,
INTERNAL_ENCRYPTION_CONTEXT_HEADER,
INTERNAL_ENCRYPTION_TAG_HEADER,
INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER,
MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER,
MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER,
MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER,
];
let needs_normalization = metadata.keys().any(|key| {
CANONICAL_KEYS
.iter()
.any(|canonical| key != canonical && key.eq_ignore_ascii_case(canonical))
});
if !needs_normalization {
return Ok(Cow::Borrowed(metadata));
}
let mut normalized = metadata.clone();
for canonical in CANONICAL_KEYS {
let mut matching_values = metadata
.iter()
.filter_map(|(key, value)| key.eq_ignore_ascii_case(canonical).then_some(value));
let Some(value) = matching_values.next() else {
continue;
};
if matching_values.any(|candidate| candidate != value) {
return Err(ConflictingEncryptionMetadata { key: canonical });
}
if !normalized.contains_key(*canonical) {
normalized.insert((*canonical).to_string(), value.clone());
}
}
Ok(Cow::Owned(normalized))
}
/// Turn the material of an encrypted write into the metadata stored with the object.
///
/// The scheme is recorded here. How the key itself is recorded belongs to the mode
/// that formed it, in `direct.rs` and `object.rs`, next to the code that reads it back.
pub fn encryption_material_to_metadata(material: &EncryptionMaterial) -> Result<HashMap<String, String>, SseError> {
let mut metadata = HashMap::new();
match material.sse_type {
SSEType::SseC => {
record_ssec_scheme(&mut metadata, material);
// A `rio-v2` build seals an object key instead, so a Direct key there is a
// multipart session created without its customer key.
if material.key_kind == EncryptionKeyKind::Direct {
record_customer_key_nonce(&mut metadata, material.base_nonce);
}
#[cfg(feature = "rio-v2")]
if let Some(sealed) = &material.managed_sealed_key {
record_sealed_customer_key(&mut metadata, sealed);
}
}
SSEType::SseS3 | SSEType::SseKms => {
let wrapped_data_key = material
.encrypted_data_key
.as_deref()
.ok_or_else(|| SseError::internal("managed SSE materials must carry an encrypted data key"))?;
record_managed_scheme(&mut metadata, material);
if material.key_kind == EncryptionKeyKind::Direct {
record_wrapped_data_key(&mut metadata, wrapped_data_key, material.base_nonce, material.algorithm.as_str());
}
#[cfg(feature = "rio-v2")]
record_data_key_in_minio_slots(&mut metadata, material, wrapped_data_key);
}
}
Ok(metadata)
}
/// SSE-C: the scheme, the fingerprint of the customer key, and the plaintext size.
fn record_ssec_scheme(metadata: &mut HashMap<String, String>, material: &EncryptionMaterial) {
metadata.insert(
"x-amz-server-side-encryption".to_string(),
material.server_side_encryption.as_str().to_string(),
);
metadata.insert(
"x-amz-server-side-encryption-customer-algorithm".to_string(),
material.algorithm.as_str().to_string(),
);
if let Some(customer_key_md5) = &material.customer_key_md5 {
metadata.insert("x-amz-server-side-encryption-customer-key-md5".to_string(), customer_key_md5.to_string());
}
if let Some(original_size) = material.original_size {
metadata.insert(SSEC_ORIGINAL_SIZE_HEADER.to_string(), original_size.to_string());
}
}
/// Managed SSE: the scheme, the key the data key is wrapped under, the plaintext
/// size, and the encryption context of the request.
fn record_managed_scheme(metadata: &mut HashMap<String, String>, material: &EncryptionMaterial) {
metadata.insert(
"x-amz-server-side-encryption".to_string(),
material.server_side_encryption.as_str().to_string(),
);
let internal_key_id = material.kms_key_id.clone().unwrap_or_else(|| "default".to_string());
metadata.insert(INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), internal_key_id.clone());
if matches!(material.sse_type, SSEType::SseKms) {
metadata.insert("x-amz-server-side-encryption-aws-kms-key-id".to_string(), internal_key_id);
}
if let Some(original_size) = material.original_size {
metadata.insert(INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(), original_size.to_string());
}
if let Some(kms_context) = &material.managed_kms_context
&& !kms_context.is_empty()
{
if let Ok(serialized) = serde_json::to_string(kms_context) {
metadata.insert(INTERNAL_ENCRYPTION_CONTEXT_HEADER.to_string(), serialized);
}
if matches!(material.sse_type, SSEType::SseKms)
&& let Ok(encoded) = encode_minio_kms_context(kms_context)
{
metadata.insert(MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER.to_string(), encoded);
}
}
}
/// Strip source encryption metadata before constructing metadata for a copy destination.
///
/// Encryption metadata describes the physical source representation and must never be
/// inherited by a plaintext destination or by a destination using a different key.
pub fn strip_managed_encryption_metadata(metadata: &mut HashMap<String, String>) {
const KEYS: [&str; 21] = [
"x-amz-server-side-encryption",
"x-amz-server-side-encryption-aws-kms-key-id",
"x-amz-server-side-encryption-customer-algorithm",
"x-amz-server-side-encryption-customer-key-md5",
SSEC_ORIGINAL_SIZE_HEADER,
INTERNAL_ENCRYPTION_KEY_ID_HEADER,
INTERNAL_ENCRYPTION_ALGORITHM_HEADER,
INTERNAL_ENCRYPTION_IV_HEADER,
INTERNAL_ENCRYPTION_TAG_HEADER,
INTERNAL_ENCRYPTION_KEY_HEADER,
INTERNAL_ENCRYPTION_CONTEXT_HEADER,
INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER,
MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER,
MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER,
MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER,
];
for key in KEYS.iter() {
metadata.remove(*key);
}
rustfs_utils::http::metadata_compat::remove_str(metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
}
pub fn mark_encrypted_multipart_metadata(metadata: &mut HashMap<String, String>) {
metadata.insert(MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER.to_string(), String::new());
}
/// True when the stored metadata marks the object as SSE-C protected.
///
/// RustFS persists the public `x-amz-server-side-encryption-customer-algorithm`
/// key on SSE-C writes. MinIO persists only its internal sealed-key slot and
/// synthesizes the public header onto responses (backlog#1638), so the slot is
/// the only durable evidence on a MinIO-written object. That shape is readable
/// solely through the rio-v2 sealed-object-key path, so it participates in
/// detection only in builds that can actually read it — default builds keep
/// the resolver's fail-closed diagnosis instead of demanding customer keys
/// they could not use.
pub fn stored_ssec_metadata(metadata: &HashMap<String, String>) -> bool {
metadata.contains_key("x-amz-server-side-encryption-customer-algorithm") || minio_sealed_ssec_metadata(metadata)
}
/// True when the object carries MinIO's SSE-C sealed-key slot — the shape that
/// stores no customer-key MD5 and proves the key through the AEAD unseal
/// instead.
pub(crate) fn minio_sealed_ssec_metadata(metadata: &HashMap<String, String>) -> bool {
cfg!(feature = "rio-v2") && metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER)
}
/// Recodes a stored MinIO KMS context value (base64-wrapped JSON) into the
/// plain-JSON form RustFS stores under [`INTERNAL_ENCRYPTION_CONTEXT_HEADER`].
///
/// Injected into the shared [`normalize_managed_metadata`] because the shared
/// crate carries no JSON codec; any decode failure returns `None`, which skips
/// the context mapping exactly like the historical inline `if let Ok` chain.
pub(crate) fn recode_minio_kms_context(value: &str) -> Option<String> {
let decoded = BASE64_STANDARD.decode_to_vec(value).ok()?;
let context = serde_json::from_slice::<HashMap<String, String>>(&decoded).ok()?;
serde_json::to_string(&context).ok()
}
-67
View File
@@ -1,67 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The internal keying modes: how an object's key is formed and how it is stored.
pub(crate) mod context;
pub(crate) mod direct;
pub(crate) mod material;
pub(crate) mod metadata;
#[cfg(feature = "rio-v2")]
pub(crate) mod object;
pub(crate) mod stored;
use crate::core::keying::material::{EncryptionKeyKind, ManagedSealedKey};
/// The key a new object is encrypted with, in the mode this build writes.
pub(crate) struct NewObjectKey {
/// Key handed to the stream cipher.
pub(crate) key_bytes: [u8; 32],
/// Base nonce the stream cipher derives its block nonces from.
pub(crate) base_nonce: [u8; 12],
/// The mode `key_bytes` belongs to.
pub(crate) key_kind: EncryptionKeyKind,
/// The sealed form to store, in a mode that seals a separate object key.
pub(crate) sealed_key: Option<ManagedSealedKey>,
}
/// A managed-SSE data key as a stored object recorded it, still wrapped.
pub(crate) struct StoredDataKey {
/// The data key, wrapped by the provider that issued it.
pub(crate) wrapped: Vec<u8>,
/// The cipher the object recorded.
pub(crate) algorithm: String,
/// What the data key is to the object once it is unwrapped.
pub(crate) mode: StoredKeyMode,
}
/// The mode a stored key was recorded in.
pub(crate) enum StoredKeyMode {
/// The data key is the object's key, under the nonce stored beside it.
Direct { base_nonce: [u8; 12] },
/// The data key unseals the object's key.
#[cfg(feature = "rio-v2")]
Object { sealed_key: ManagedSealedKey },
}
// The mode new objects are written in. Every build reads every mode it can
// recognize; writing picks exactly one, and this is the only place that says which.
#[cfg(not(feature = "rio-v2"))]
pub(crate) use direct::{
SESSION_KEY_NEEDS_CUSTOMER_KEY, new_key_under_customer_key, new_key_under_data_key, new_session_key_under_customer_key,
};
#[cfg(feature = "rio-v2")]
pub(crate) use object::{
SESSION_KEY_NEEDS_CUSTOMER_KEY, new_key_under_customer_key, new_key_under_data_key, new_session_key_under_customer_key,
};
-436
View File
@@ -1,436 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Object keying: a derived object key, sealed in the MinIO-compatible format.
use crate::core::keying::material::{EncryptionKeyKind, EncryptionMaterial, ManagedSealedKey};
use crate::core::keying::{NewObjectKey, StoredDataKey, StoredKeyMode};
use crate::error::SseError;
use aes_gcm::aead::{Aead, KeyInit, Payload};
use aes_gcm::{Aes256Gcm, Nonce};
use base64_simd::STANDARD as BASE64_STANDARD;
use chacha20poly1305::ChaCha20Poly1305;
use hmac::{Hmac, Mac};
use rand::RngExt;
use rustfs_kms::DataKey;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_IV_HEADER, INTERNAL_ENCRYPTION_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER, MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER, SSEType,
};
use sha2::Sha256;
use std::collections::HashMap;
pub(crate) const MINIO_INTERNAL_ENCRYPTION_SEAL_ALGORITHM: &str = "DAREv2-HMAC-SHA256";
const DARE_VERSION_20: u8 = 0x20;
pub(crate) const DARE_CIPHER_AES_256_GCM: u8 = 0x00;
pub(crate) const DARE_CIPHER_CHACHA20_POLY1305: u8 = 0x01;
const DARE_HEADER_SIZE: usize = 16;
const DARE_TAG_SIZE: usize = 16;
pub(crate) const SEALED_KEY_IV_SIZE: usize = 32;
pub(crate) const SEALED_KEY_SIZE: usize = DARE_HEADER_SIZE + 32 + DARE_TAG_SIZE;
const OBJECT_KEY_DERIVATION_CONTEXT: &[u8] = b"object-encryption-key generation";
pub(crate) type HmacSha256 = Hmac<Sha256>;
fn managed_sse_domain(sse_type: SSEType) -> &'static str {
match sse_type {
SSEType::SseS3 => "SSE-S3",
SSEType::SseKms => "SSE-KMS",
SSEType::SseC => "SSE-C",
}
}
pub(crate) fn is_supported_sealed_object_key_cipher(cipher: u8) -> bool {
matches!(cipher, DARE_CIPHER_AES_256_GCM | DARE_CIPHER_CHACHA20_POLY1305)
}
fn decrypt_sealed_object_key_payload(sealing_key: [u8; 32], header: &[u8], sealed_key: &[u8]) -> Result<Vec<u8>, SseError> {
let nonce = &header[4..16];
let ciphertext = &sealed_key[DARE_HEADER_SIZE..];
let aad = &header[..4];
match header[1] {
DARE_CIPHER_AES_256_GCM => {
let cipher = Aes256Gcm::new_from_slice(&sealing_key)
.map_err(|err| SseError::internal(format!("Invalid AES-GCM sealing key: {err}")))?;
let nonce = Nonce::try_from(nonce).map_err(|_| SseError::internal("Invalid sealed object-key package nonce"))?;
cipher.decrypt(&nonce, Payload { msg: ciphertext, aad })
}
DARE_CIPHER_CHACHA20_POLY1305 => {
let cipher = ChaCha20Poly1305::new_from_slice(&sealing_key)
.map_err(|err| SseError::internal(format!("Invalid ChaCha20-Poly1305 sealing key: {err}")))?;
let nonce = chacha20poly1305::Nonce::try_from(nonce)
.map_err(|_| SseError::internal("Invalid sealed object-key package nonce"))?;
cipher.decrypt(&nonce, Payload { msg: ciphertext, aad })
}
_ => return Err(SseError::internal("Unsupported sealed object-key DARE header")),
}
.map_err(|err| SseError::internal(format!("Failed to unseal object key: {err}")))
}
fn canonical_sse_path(bucket: &str, object: &str) -> String {
let bucket = bucket.trim_matches('/');
let object = object.trim_matches('/');
if object.is_empty() {
bucket.to_string()
} else if bucket.is_empty() {
object.to_string()
} else {
format!("{bucket}/{object}")
}
}
pub(crate) fn derive_object_key(external_key: [u8; 32]) -> Result<[u8; 32], SseError> {
let mut random = [0u8; 32];
rand::rng().fill(&mut random);
let mut mac = HmacSha256::new_from_slice(&external_key)
.map_err(|err| SseError::internal(format!("Invalid HMAC key for object-key derivation: {err}")))?;
mac.update(OBJECT_KEY_DERIVATION_CONTEXT);
mac.update(&random);
let mut object_key = [0u8; 32];
object_key.copy_from_slice(mac.finalize().into_bytes().as_slice());
Ok(object_key)
}
fn derive_sealing_key(
external_key: [u8; 32],
iv: [u8; SEALED_KEY_IV_SIZE],
domain: &str,
bucket: &str,
object: &str,
) -> Result<[u8; 32], SseError> {
let mut mac = HmacSha256::new_from_slice(&external_key)
.map_err(|err| SseError::internal(format!("Invalid HMAC key for sealing-key derivation: {err}")))?;
mac.update(&iv);
mac.update(domain.as_bytes());
mac.update(MINIO_INTERNAL_ENCRYPTION_SEAL_ALGORITHM.as_bytes());
mac.update(canonical_sse_path(bucket, object).as_bytes());
let mut sealing_key = [0u8; 32];
sealing_key.copy_from_slice(mac.finalize().into_bytes().as_slice());
Ok(sealing_key)
}
pub(crate) fn seal_object_key(
object_key: [u8; 32],
external_key: [u8; 32],
sse_type: SSEType,
bucket: &str,
object: &str,
) -> Result<ManagedSealedKey, SseError> {
let mut iv = [0u8; SEALED_KEY_IV_SIZE];
rand::rng().fill(&mut iv);
let sealing_key = derive_sealing_key(external_key, iv, managed_sse_domain(sse_type), bucket, object)?;
let mut header = [0u8; DARE_HEADER_SIZE];
header[0] = DARE_VERSION_20;
header[1] = DARE_CIPHER_AES_256_GCM;
header[2..4].copy_from_slice(&(31u16).to_le_bytes());
let mut stream_nonce = [0u8; 12];
rand::rng().fill(&mut stream_nonce);
stream_nonce[0] |= 0x80;
header[4..16].copy_from_slice(&stream_nonce);
let cipher =
Aes256Gcm::new_from_slice(&sealing_key).map_err(|err| SseError::internal(format!("Invalid sealing key: {err}")))?;
let nonce =
Nonce::try_from(stream_nonce.as_slice()).map_err(|_| SseError::internal("Invalid sealed object-key stream nonce"))?;
let ciphertext = cipher
.encrypt(
&nonce,
aes_gcm::aead::Payload {
msg: &object_key,
aad: &header[..4],
},
)
.map_err(|err| SseError::internal(format!("Failed to seal object key: {err}")))?;
let mut sealed_key = [0u8; SEALED_KEY_SIZE];
sealed_key[..DARE_HEADER_SIZE].copy_from_slice(&header);
sealed_key[DARE_HEADER_SIZE..].copy_from_slice(&ciphertext);
Ok(ManagedSealedKey { iv, sealed_key })
}
pub(crate) fn unseal_object_key(
sealed: &ManagedSealedKey,
external_key: [u8; 32],
sse_type: SSEType,
bucket: &str,
object: &str,
) -> Result<[u8; 32], SseError> {
let header = &sealed.sealed_key[..DARE_HEADER_SIZE];
if header[0] != DARE_VERSION_20 || !is_supported_sealed_object_key_cipher(header[1]) {
return Err(SseError::internal("Unsupported sealed object-key DARE header"));
}
if u16::from_le_bytes([header[2], header[3]]) != 31 || header[4] & 0x80 == 0 {
return Err(SseError::internal("Invalid sealed object-key payload header"));
}
let sealing_key = derive_sealing_key(external_key, sealed.iv, managed_sse_domain(sse_type), bucket, object)?;
let plaintext = decrypt_sealed_object_key_payload(sealing_key, header, &sealed.sealed_key)?;
let object_key: [u8; 32] = plaintext
.as_slice()
.try_into()
.map_err(|_| SseError::internal("Sealed object key must decrypt to 32 bytes"))?;
Ok(object_key)
}
fn try_decode_minio_sealed_key(bytes: &str) -> Result<Option<[u8; SEALED_KEY_SIZE]>, SseError> {
let decoded = BASE64_STANDARD
.decode_to_vec(bytes)
.map_err(|e| SseError::internal(format!("Failed to decode sealed object key: {e}")))?;
match decoded.as_slice().try_into() {
Ok(sealed_key) => Ok(Some(sealed_key)),
Err(_) => Ok(None),
}
}
fn try_decode_minio_sealing_iv(bytes: &str) -> Result<Option<[u8; SEALED_KEY_IV_SIZE]>, SseError> {
let decoded = BASE64_STANDARD
.decode_to_vec(bytes)
.map_err(|e| SseError::internal(format!("Failed to decode sealing IV: {e}")))?;
match decoded.as_slice().try_into() {
Ok(iv) => Ok(Some(iv)),
Err(_) => Ok(None),
}
}
pub(crate) fn is_legacy_rustfs_managed_metadata(metadata: &HashMap<String, String>) -> bool {
metadata.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER)
&& metadata.contains_key(INTERNAL_ENCRYPTION_IV_HEADER)
&& !metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER)
&& !metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER)
}
/// Infer the managed SSE scheme from the MinIO sealed-key slot that is present.
///
/// Returns `None` when no managed MinIO slot is present, which keeps callers on
/// their fail-closed path. SSE-C is not a managed scheme and is handled by the
/// SSE-C read path, so its slot is not considered here.
pub(crate) fn infer_minio_managed_sse_type(metadata: &HashMap<String, String>) -> Option<SSEType> {
if metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER) {
Some(SSEType::SseS3)
} else if metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER) {
Some(SSEType::SseKms)
} else {
None
}
}
pub(crate) fn parse_minio_managed_sealed_key(
metadata: &HashMap<String, String>,
sse_type: SSEType,
) -> Result<Option<ManagedSealedKey>, SseError> {
let algorithm = match metadata.get(MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER) {
Some(algorithm) => algorithm,
None => return Ok(None),
};
if algorithm != MINIO_INTERNAL_ENCRYPTION_SEAL_ALGORITHM {
return Ok(None);
}
let iv = match metadata.get(MINIO_INTERNAL_ENCRYPTION_IV_HEADER) {
Some(iv) => match try_decode_minio_sealing_iv(iv)? {
Some(iv) => iv,
None => return Ok(None),
},
None => return Ok(None),
};
let header_key = match sse_type {
SSEType::SseS3 => MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
SSEType::SseKms => MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
SSEType::SseC => MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
};
let sealed_key = match metadata.get(header_key) {
Some(value) => match try_decode_minio_sealed_key(value)? {
Some(sealed_key) => sealed_key,
None => return Ok(None),
},
None => return Ok(None),
};
Ok(Some(ManagedSealedKey { iv, sealed_key }))
}
/// SSE-C: record the object key sealed under the customer key.
pub(crate) fn record_sealed_customer_key(metadata: &mut HashMap<String, String>, sealed: &ManagedSealedKey) {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(sealed.iv),
);
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(),
MINIO_INTERNAL_ENCRYPTION_SEAL_ALGORITHM.to_string(),
);
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(sealed.sealed_key),
);
}
/// Managed SSE: record the wrapped data key in MinIO's slots.
///
/// With a sealed object key the slots hold what MinIO itself writes. Without one
/// the key was formed in Direct mode, and the slots mirror its wrapped data key and
/// nonce: [`parse_minio_managed_sealed_key`] turns that shape down by its length, so
/// the object is read back in Direct mode.
pub(crate) fn record_data_key_in_minio_slots(
metadata: &mut HashMap<String, String>,
material: &EncryptionMaterial,
wrapped_data_key: &[u8],
) {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(),
MINIO_INTERNAL_ENCRYPTION_SEAL_ALGORITHM.to_string(),
);
if let Some(kms_key_id) = &material.kms_key_id {
metadata.insert(MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER.to_string(), kms_key_id.to_string());
}
let encoded_data_key = BASE64_STANDARD.encode_to_string(wrapped_data_key);
match &material.managed_sealed_key {
Some(sealed) => {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(sealed.iv),
);
match material.sse_type {
SSEType::SseS3 => {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(sealed.sealed_key),
);
}
SSEType::SseKms => {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(sealed.sealed_key),
);
}
SSEType::SseC => {}
}
metadata.insert(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER.to_string(), encoded_data_key);
}
None => {
metadata.insert(
MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(),
BASE64_STANDARD.encode_to_string(material.base_nonce),
);
match material.sse_type {
SSEType::SseS3 => {
metadata.insert(MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER.to_string(), encoded_data_key);
}
SSEType::SseKms => {
metadata.insert(MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER.to_string(), encoded_data_key.clone());
metadata.insert(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER.to_string(), encoded_data_key);
}
SSEType::SseC => {}
}
}
}
}
/// Managed SSE: read back the wrapped data key that seals `sealed_key`.
///
/// `normalized_metadata` is `metadata` with the managed keys normalized.
pub(crate) fn read_sealed_data_key(
metadata: &HashMap<String, String>,
normalized_metadata: &HashMap<String, String>,
sealed_key: ManagedSealedKey,
) -> Result<StoredDataKey, SseError> {
let encrypted_key_b64 = normalized_metadata
.get(INTERNAL_ENCRYPTION_KEY_HEADER)
.or_else(|| metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER))
.ok_or_else(|| SseError::internal("Missing encrypted key in metadata"))?;
let wrapped = BASE64_STANDARD
.decode_to_vec(encrypted_key_b64)
.map_err(|e| SseError::internal(format!("Failed to decode encrypted key: {e}")))?;
Ok(StoredDataKey {
wrapped,
algorithm: normalized_metadata
.get(INTERNAL_ENCRYPTION_ALGORITHM_HEADER)
.cloned()
.unwrap_or_else(|| "AES256".to_string()),
mode: StoredKeyMode::Object { sealed_key },
})
}
/// Whether a multipart session needs the customer key when it is created.
///
/// Object mode seals the session's object key under the customer key, so the key
/// has to be present, and valid, at that point.
pub(crate) const SESSION_KEY_NEEDS_CUSTOMER_KEY: bool = true;
/// Derive a fresh object key from `external_key` and seal it under the same key.
fn new_sealed_object_key(external_key: [u8; 32], sse_type: SSEType, bucket: &str, key: &str) -> Result<NewObjectKey, SseError> {
let object_key = derive_object_key(external_key)?;
let sealed_key = seal_object_key(object_key, external_key, sse_type, bucket, key)?;
Ok(NewObjectKey {
key_bytes: object_key,
base_nonce: [0; 12],
key_kind: EncryptionKeyKind::Object,
sealed_key: Some(sealed_key),
})
}
/// SSE-C object: an object key sealed under the customer key.
pub(crate) fn new_key_under_customer_key(customer_key: [u8; 32], bucket: &str, key: &str) -> Result<NewObjectKey, SseError> {
new_sealed_object_key(customer_key, SSEType::SseC, bucket, key)
}
/// SSE-C multipart session: an object key sealed under the customer key.
///
/// A session created without its customer key has nothing to seal and records a
/// placeholder instead.
pub(crate) fn new_session_key_under_customer_key(
customer_key: Option<[u8; 32]>,
bucket: &str,
key: &str,
) -> Result<NewObjectKey, SseError> {
match customer_key {
Some(customer_key) => new_sealed_object_key(customer_key, SSEType::SseC, bucket, key),
None => Ok(NewObjectKey {
key_bytes: [0; 32],
base_nonce: [0; 12],
key_kind: EncryptionKeyKind::Direct,
sealed_key: None,
}),
}
}
/// Managed SSE: an object key sealed under the data key.
pub(crate) fn new_key_under_data_key(
data_key: &DataKey,
sse_type: SSEType,
bucket: &str,
key: &str,
) -> Result<NewObjectKey, SseError> {
new_sealed_object_key(data_key.plaintext_key, sse_type, bucket, key)
}
-253
View File
@@ -1,253 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Reading: recognize how a stored object was keyed, and recover its key.
//!
//! What an object recorded is the only source of truth here. The scheme a request
//! advertises and whether a KMS is running now are write policy and runtime state;
//! neither says how an object written earlier was keyed.
use crate::core::keying::context::{build_object_encryption_context, decode_minio_kms_context};
use crate::core::keying::direct::{read_stored_ssec_nonce, read_wrapped_data_key};
use crate::core::keying::material::EncryptionKeyKind;
use crate::core::keying::metadata::recode_minio_kms_context;
#[cfg(feature = "rio-v2")]
use crate::core::keying::object::{
infer_minio_managed_sse_type, is_legacy_rustfs_managed_metadata, parse_minio_managed_sealed_key, read_sealed_data_key,
unseal_object_key,
};
use crate::core::keying::{StoredDataKey, StoredKeyMode};
use crate::error::SseError;
use crate::host::SseHost;
use crate::kms::local::is_local_sse_dek;
#[cfg(feature = "rio-v2")]
use crate::kms::local::is_rustfs_managed_data_key;
#[cfg(feature = "rio-v2")]
use crate::kms::provider::decrypt_unenveloped_sealed_dek;
use crate::kms::provider::{SseDekProvider, bare_data_key_kms_provider, get_local_sse_dek_provider, kms_sse_dek_provider};
use rustfs_kms::is_data_key_envelope;
use rustfs_utils::http::headers::AMZ_ENCRYPTION_KMS;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_KEY_ID_HEADER, SSEType, contains_managed_encryption_metadata, normalize_managed_metadata,
};
use std::collections::HashMap;
use std::sync::Arc;
/// The key a stored object is decrypted with, recovered from what the object recorded.
pub(crate) struct OpenedObjectKey {
/// Key handed to the stream cipher.
pub(crate) key_bytes: [u8; 32],
/// Base nonce the stream cipher derives its block nonces from.
pub(crate) base_nonce: [u8; 12],
/// The mode the object was written in.
pub(crate) key_kind: EncryptionKeyKind,
}
/// The managed scheme a stored object was written under, or `None` when this build
/// cannot establish it.
fn stored_managed_sse_type(metadata: &HashMap<String, String>) -> Option<SSEType> {
match metadata.get("x-amz-server-side-encryption").map(String::as_str) {
Some(AMZ_ENCRYPTION_KMS) => Some(SSEType::SseKms),
Some(_) => Some(SSEType::SseS3),
// MinIO never persists the public scheme header: `crypto.S3.CreateMetadata`
// writes only the `X-Minio-Internal-*` family and the public header is
// synthesized onto the response by `DecryptObjectInfo`. Requiring it here
// is what made every MinIO-encrypted object unreadable (backlog#1638).
//
// Inferring from the sealed-key slot is self-consistent by construction:
// the slot decides which header the unseal reads AND which domain string
// the sealing key is derived under, so a scheme that disagrees with the
// slot cannot silently derive a wrong key — it finds no key at all.
// Inferring from the KMS key id would NOT be safe: MinIO writes
// `-S3-Kms-Key-Id` on SSE-S3 objects too.
//
// An object whose scheme cannot be established stays fail-closed, and
// deliberately not through an error raised here: the read plan independently
// classifies the object as encrypted from its markers and refuses to serve
// it without material, so it never degrades into a plaintext read.
#[cfg(feature = "rio-v2")]
None => infer_minio_managed_sse_type(metadata),
// Without the rio-v2 reader there is no MinIO-format read path to serve
// such an object with, so it stays on the fail-closed branch.
#[cfg(not(feature = "rio-v2"))]
None => None,
}
}
/// SSE-C: recover the object's key from the customer key.
///
/// A sealed object key recorded in MinIO's slot is unsealed under the customer key;
/// otherwise the customer key is the object's key, under the nonce the object recorded.
pub(crate) fn open_under_customer_key(
metadata: &HashMap<String, String>,
customer_key: [u8; 32],
bucket: &str,
key: &str,
) -> Result<OpenedObjectKey, SseError> {
#[cfg(feature = "rio-v2")]
if let Some(sealed_key) = parse_minio_managed_sealed_key(metadata, SSEType::SseC)? {
return Ok(OpenedObjectKey {
key_bytes: unseal_object_key(&sealed_key, customer_key, SSEType::SseC, bucket, key)?,
base_nonce: [0; 12],
key_kind: EncryptionKeyKind::Object,
});
}
Ok(OpenedObjectKey {
key_bytes: customer_key,
base_nonce: read_stored_ssec_nonce(metadata, bucket, key),
key_kind: EncryptionKeyKind::Direct,
})
}
/// A stored managed-SSE object, as recognized from its metadata alone.
pub(crate) struct StoredManagedKey<'a> {
/// The object's metadata as stored.
metadata: &'a HashMap<String, String>,
/// The same metadata with the managed keys normalized.
normalized_metadata: HashMap<String, String>,
/// The scheme the object was written under.
pub(crate) sse_type: SSEType,
/// The KMS key id the object recorded, if it recorded one.
pub(crate) recorded_kms_key_id: Option<String>,
}
/// Recognize a stored managed-SSE object.
///
/// `None` means the object carries no managed-SSE metadata, or carries some whose
/// scheme this build cannot establish.
pub(crate) fn recognize_stored_managed_key(metadata: &HashMap<String, String>) -> Option<StoredManagedKey<'_>> {
if !contains_managed_encryption_metadata(metadata) {
return None;
}
let sse_type = stored_managed_sse_type(metadata)?;
let normalized_metadata = normalize_managed_metadata(metadata, Some(recode_minio_kms_context));
let recorded_kms_key_id = normalized_metadata
.get(INTERNAL_ENCRYPTION_KEY_ID_HEADER)
.or_else(|| metadata.get("x-amz-server-side-encryption-aws-kms-key-id"))
.cloned();
Some(StoredManagedKey {
metadata,
normalized_metadata,
sse_type,
recorded_kms_key_id,
})
}
/// Managed SSE: unwrap the stored data key and recover the object's key from it.
///
/// `kms_key_id` is the key the read was authorized against. Returns the object's key
/// together with the algorithm the object recorded.
pub(crate) async fn open_under_stored_data_key(
host: &dyn SseHost,
bucket: &str,
key: &str,
stored: &StoredManagedKey<'_>,
kms_key_id: &str,
) -> Result<(OpenedObjectKey, String), SseError> {
let record = read_stored_data_key(host, stored).await?;
let data_key = unwrap_stored_data_key(host, bucket, key, stored, &record, kms_key_id).await?;
let opened = match record.mode {
StoredKeyMode::Direct { base_nonce } => OpenedObjectKey {
key_bytes: data_key,
base_nonce,
key_kind: EncryptionKeyKind::Direct,
},
#[cfg(feature = "rio-v2")]
StoredKeyMode::Object { sealed_key } => OpenedObjectKey {
key_bytes: unseal_object_key(&sealed_key, data_key, stored.sse_type, bucket, key)?,
base_nonce: [0u8; 12],
key_kind: EncryptionKeyKind::Object,
},
};
Ok((opened, record.algorithm))
}
/// Read the stored data key in the mode the object recorded.
///
/// This is the recognition order: a mode listed earlier claims the object first, and
/// Direct takes whatever no other mode recognizes. A mode this build cannot read is
/// not listed, so its objects fail in the Direct reader instead of being misread.
async fn read_stored_data_key(host: &dyn SseHost, stored: &StoredManagedKey<'_>) -> Result<StoredDataKey, SseError> {
#[cfg(feature = "rio-v2")]
if let Some(sealed_key) = parse_minio_managed_sealed_key(stored.metadata, stored.sse_type)? {
return read_sealed_data_key(stored.metadata, &stored.normalized_metadata, sealed_key);
}
read_wrapped_data_key(host, &stored.normalized_metadata).await
}
/// Unwrap the stored data key with the provider its own bytes identify.
async fn unwrap_stored_data_key(
host: &dyn SseHost,
bucket: &str,
key: &str,
stored: &StoredManagedKey<'_>,
record: &StoredDataKey,
kms_key_id: &str,
) -> Result<[u8; 32], SseError> {
let kms_context = if matches!(stored.sse_type, SSEType::SseKms) {
decode_minio_kms_context(stored.metadata)?
} else {
None
};
let object_context = build_object_encryption_context(bucket, key, kms_context.as_ref());
let wrapped = record.wrapped.as_slice();
// Persisted wrapping format is the read-side source of truth. The
// advertised SSE scheme and current KMS availability are write policy
// and runtime state, neither of which identifies the historical provider.
//
// The one shape that cannot name its provider is a backend-native blob
// stored without an envelope. It is offered to the KMS only when the
// local provider could not have written those bytes and the active
// backend is one whose released versions persisted them that way.
let bare_kms = if is_data_key_envelope(wrapped) || is_local_sse_dek(wrapped) {
None
} else {
bare_data_key_kms_provider(host).await
};
// A MinIO sealed key alone does not mean MinIO wrote the object: RustFS's own
// writer fills MinIO's metadata slots too, while still storing a RustFS
// envelope in them, so neither the slot nor the header name distinguishes the
// two. The data key's own shape does. RustFS envelopes are strictly-parsed
// JSON; MinIO's builtin-KMS ciphertext is opaque bytes that match neither, so
// recognizing RustFS positively — and treating only the remainder as MinIO —
// keeps a RustFS envelope from ever reaching MinIO's decoder.
#[cfg(feature = "rio-v2")]
if matches!(record.mode, StoredKeyMode::Object { .. }) && !is_rustfs_managed_data_key(wrapped) {
return decrypt_unenveloped_sealed_dek(bare_kms, get_local_sse_dek_provider(), wrapped, kms_key_id, &object_context)
.await;
}
let provider: Arc<dyn SseDekProvider> = if is_data_key_envelope(wrapped) {
kms_sse_dek_provider(host).await?
} else if let Some(kms_provider) = bare_kms {
kms_provider
} else {
get_local_sse_dek_provider()?
};
#[cfg(feature = "rio-v2")]
if is_legacy_rustfs_managed_metadata(&stored.normalized_metadata) {
return provider.decrypt_legacy_sse_dek(wrapped, kms_key_id, &object_context).await;
}
provider.decrypt_sse_dek(wrapped, kms_key_id, &object_context).await
}
-23
View File
@@ -1,23 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! SSE key resolution: the S3-defined schemes, the internal keying modes, and the entry points that join them.
pub(crate) mod audit;
pub(crate) mod keying;
pub(crate) mod read;
pub(crate) mod request;
pub(crate) mod rewrap;
pub(crate) mod s3;
pub(crate) mod write;
-303
View File
@@ -1,303 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Read entry points: resolve the decryption material, or the response headers, for a stored object.
use crate::core::audit::{record_managed_kms_outcome, stored_envelope_master_key_version};
use crate::core::keying::material::DecryptionMaterial;
use crate::core::keying::metadata::stored_ssec_metadata;
use crate::core::keying::stored::{
StoredManagedKey, open_under_customer_key, open_under_stored_data_key, recognize_stored_managed_key,
};
use crate::core::request::SseKmsPrincipal;
use crate::core::s3::authorization::authorize_sse_kms_key;
use crate::core::s3::headers::{SseReadResponseHeaders, managed_sse_public_header};
use crate::core::s3::scheme::resolve_stored_kms_key_id;
use crate::core::s3::ssec::{SsecParams, require_matching_ssec_key, stored_ssec_algorithm, validate_ssec_params};
use crate::error::SseError;
use crate::host::SseHost;
use rustfs_policy::policy::action::KmsAction;
use rustfs_utils::http::headers::AMZ_ENCRYPTION_AES;
use rustfs_utils::http::object_encryption_keys::{SSEType, contains_managed_encryption_metadata, stored_managed_encryption_key};
use std::collections::HashMap;
impl std::fmt::Debug for DecryptionRequest<'_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("DecryptionRequest").finish_non_exhaustive()
}
}
/// Request parameters for unified decryption
pub struct DecryptionRequest<'a> {
/// Bucket name
pub bucket: &'a str,
/// Object key
pub key: &'a str,
/// Object metadata containing encryption headers
pub metadata: &'a HashMap<String, String>,
/// SSE-C key (Base64-encoded) - required if object was encrypted with SSE-C
pub sse_customer_key: Option<&'a String>,
/// SSE-C key MD5 (Base64-encoded) - required if object was encrypted with SSE-C
pub sse_customer_key_md5: Option<&'a String>,
/// Caller the SSE-KMS key usage is authorized as. `None` marks an internal caller.
pub principal: Option<&'a SseKmsPrincipal>,
}
/// **Core API**: Apply decryption based on stored metadata
///
/// This function automatically detects the encryption type from metadata:
/// - SSE-C if customer key is provided
/// - SSE-S3/SSE-KMS if managed encryption metadata is found
/// - None if object is not encrypted
///
/// # Arguments
/// * `request` - Decryption request with metadata and optional customer key
///
/// # Returns
/// * `Ok(Some(material))` - Decryption should be applied with the returned material
/// * `Ok(None)` - Object is not encrypted
/// * `Err` - Decryption configuration error or key mismatch
///
/// # Example
/// ```rust,ignore
/// let request = DecryptionRequest {
/// bucket: &bucket,
/// key: &key,
/// metadata: &metadata,
/// sse_customer_key: sse_customer_key.as_deref(),
/// sse_customer_key_md5: sse_customer_key_md5.as_deref(),
/// };
///
/// if let Some(material) = sse_decryption(host, request).await? {
/// content_size = material.original_size.unwrap_or(actual_size);
/// }
/// ```
pub async fn sse_decryption(host: &dyn SseHost, request: DecryptionRequest<'_>) -> Result<Option<DecryptionMaterial>, SseError> {
// Check for SSE-C encryption
if stored_ssec_metadata(request.metadata) {
let (key, key_md5) = require_matching_ssec_key(request.metadata, request.sse_customer_key, request.sse_customer_key_md5)?;
let mut material = apply_ssec_decryption_material(request.bucket, request.key, request.metadata, key, key_md5).await?;
material.customer_key_md5 = Some(key_md5.clone());
return Ok(Some(material));
}
// Check for managed SSE encryption
if contains_managed_encryption_metadata(request.metadata) {
return apply_managed_decryption_material(host, request.bucket, request.key, request.metadata, request.principal).await;
}
// No encryption detected
Ok(None)
}
/// Read-side response classification for an object whose payload the object
/// layer's encryption resolver already decrypted.
///
/// [`sse_decryption`] both unwraps the data key and derives the response
/// headers. The GET path only needs the latter — its stream comes out of the
/// object layer decrypted (see [`SseObjectEncryptionResolver`]) — so calling
/// [`sse_decryption`] there performed a second KMS `Decrypt` per request whose
/// key bytes were discarded. This function reproduces that call's header,
/// validation, authorization and audit behavior from stored metadata alone.
///
/// Contract, mirrored from [`sse_decryption`] item by item:
/// - SSE-C: missing request key/MD5 and stored/request MD5 mismatch fail with
/// the same errors and precedence, and the provided key is fully validated.
/// - Managed SSE: per-key `kms:Decrypt` authorization runs ahead of every other
/// failure mode, and the request's KMS audit summary records the same
/// scheme/key/version/outcome fields the unwrap-based path recorded. The
/// success outcome is honest because a failed unwrap aborts the read in the
/// object layer before response classification is ever reached.
/// - Objects without SSE metadata — and managed metadata whose scheme cannot
/// be established in this build — classify as `None`, exactly where
/// [`sse_decryption`] returned `None`.
pub async fn classify_sse_read_response(
host: &dyn SseHost,
request: DecryptionRequest<'_>,
) -> Result<Option<SseReadResponseHeaders>, SseError> {
if stored_ssec_metadata(request.metadata) {
let (key, key_md5) = require_matching_ssec_key(request.metadata, request.sse_customer_key, request.sse_customer_key_md5)?;
let algorithm = stored_ssec_algorithm(request.metadata);
validate_ssec_params(SsecParams {
algorithm: algorithm.to_string(),
key: key.to_string(),
key_md5: key_md5.to_string(),
})?;
return Ok(Some(SseReadResponseHeaders {
server_side_encryption: String::from(AMZ_ENCRYPTION_AES),
sse_customer_algorithm: Some(String::from(algorithm)),
sse_customer_key_md5: Some(key_md5.clone()),
ssekms_key_id: None,
}));
}
// Same scheme and key resolution as the unwrap path, so authorization and the
// response header name the same key.
let Some((stored, kms_key_id)) = resolve_stored_managed_key(host, request.metadata).await else {
return Ok(None);
};
// Ahead of every other failure mode, so a denied caller learns nothing
// about the key beyond "not yours".
if let Err(error) =
authorize_sse_kms_key(host, request.principal, stored.sse_type, KmsAction::DecryptAction, &kms_key_id).await
{
if let Some((sse_type, key_id)) = stored_managed_encryption_key(request.metadata) {
record_managed_kms_outcome(
request.principal,
sse_type,
Some(&key_id),
|| stored_envelope_master_key_version(request.metadata),
Err(&error),
);
}
return Err(error);
}
record_managed_kms_outcome(
request.principal,
stored.sse_type,
Some(&kms_key_id),
|| stored_envelope_master_key_version(request.metadata),
Ok(()),
);
Ok(Some(SseReadResponseHeaders {
server_side_encryption: managed_sse_public_header(stored.sse_type).to_string(),
sse_customer_algorithm: None,
sse_customer_key_md5: None,
// The key id was needed above to authorize the read, but an AES256
// object's wrapping key is internal: only aws:kms objects advertise it.
ssekms_key_id: matches!(stored.sse_type, SSEType::SseKms).then_some(kms_key_id),
}))
}
/// Recognize a stored managed-SSE object and name the key it is read under.
///
/// One resolution for every read: authorization, the response headers and the
/// unwrap all name the key this returns. `None` means the object carries no
/// managed-SSE metadata, or carries some whose scheme this build cannot establish.
async fn resolve_stored_managed_key<'m>(
host: &dyn SseHost,
metadata: &'m HashMap<String, String>,
) -> Option<(StoredManagedKey<'m>, String)> {
let stored = recognize_stored_managed_key(metadata)?;
let kms_key_id = resolve_stored_kms_key_id(host, stored.recorded_kms_key_id.clone()).await;
Some((stored, kms_key_id))
}
pub(crate) async fn apply_ssec_decryption_material(
bucket: &str,
key: &str,
metadata: &HashMap<String, String>,
sse_key: &str,
sse_key_md5: &str,
) -> Result<DecryptionMaterial, SseError> {
// Validate provided key
let algorithm = stored_ssec_algorithm(metadata);
let params = SsecParams {
algorithm: algorithm.to_string(),
key: sse_key.to_string(),
key_md5: sse_key_md5.to_string(),
};
let validated = validate_ssec_params(params)?;
let opened = open_under_customer_key(metadata, validated.key_bytes, bucket, key)?;
Ok(DecryptionMaterial {
sse_type: SSEType::SseC,
server_side_encryption: String::from(AMZ_ENCRYPTION_AES), // const
kms_key_id: None,
algorithm: String::from(algorithm),
customer_key_md5: None,
key_bytes: opened.key_bytes,
base_nonce: opened.base_nonce,
key_kind: opened.key_kind,
})
}
pub(crate) async fn apply_managed_decryption_material(
host: &dyn SseHost,
bucket: &str,
key: &str,
metadata: &HashMap<String, String>,
principal: Option<&SseKmsPrincipal>,
) -> Result<Option<DecryptionMaterial>, SseError> {
let result = apply_managed_decryption_material_inner(host, bucket, key, metadata, principal).await;
match &result {
// `None` means the object carries no managed-SSE metadata — SSE-C and
// plaintext objects never reach KMS and must not appear in the summary.
Ok(None) => {}
Ok(Some(material)) => record_managed_kms_outcome(
principal,
material.sse_type,
material.kms_key_id.as_deref(),
|| stored_envelope_master_key_version(metadata),
Ok(()),
),
Err(error) => {
if let Some((sse_type, key_id)) = stored_managed_encryption_key(metadata) {
record_managed_kms_outcome(
principal,
sse_type,
Some(&key_id),
|| stored_envelope_master_key_version(metadata),
Err(error),
);
}
}
}
result
}
async fn apply_managed_decryption_material_inner(
host: &dyn SseHost,
bucket: &str,
key: &str,
metadata: &HashMap<String, String>,
principal: Option<&SseKmsPrincipal>,
) -> Result<Option<DecryptionMaterial>, SseError> {
let Some((stored, kms_key_id)) = resolve_stored_managed_key(host, metadata).await else {
return Ok(None);
};
// Ahead of every other failure mode below, so a denied caller learns nothing about the
// key beyond "not yours" — not whether it is disabled, pending deletion, or unreadable.
authorize_sse_kms_key(host, principal, stored.sse_type, KmsAction::DecryptAction, &kms_key_id).await?;
let (opened, algorithm) = open_under_stored_data_key(host, bucket, key, &stored, &kms_key_id).await?;
Ok(Some(DecryptionMaterial {
sse_type: stored.sse_type,
// Synthesized from the resolved scheme rather than read back from
// metadata: a MinIO-written object has no stored scheme header, which is
// exactly why the scheme had to be inferred. MinIO synthesizes the same
// header onto its own responses.
server_side_encryption: managed_sse_public_header(stored.sse_type).to_string(),
kms_key_id: Some(kms_key_id),
algorithm,
customer_key_md5: None,
key_bytes: opened.key_bytes,
base_nonce: opened.base_nonce,
key_kind: opened.key_kind,
}))
}
-148
View File
@@ -1,148 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The caller an SSE operation is authorized as.
use crate::core::audit::{KmsRequestAudit, kms_request_audit};
#[cfg(any(test, feature = "test-util"))]
use crate::host::KmsKeyAuthorizer;
use serde_json::Value;
use std::collections::HashMap;
use std::sync::Arc;
/// Identity a data-path KMS key operation is attributed to.
///
/// Only requests that crossed the S3 authorization boundary carry one. Internal
/// callers — replication, lifecycle transition, heal, scanner — enter the object
/// layer directly and run with server credentials, so they never build a principal
/// and are exempt by construction; `None` therefore means "system principal".
#[derive(Debug, Clone)]
pub struct SseKmsPrincipal {
pub(crate) account: String,
groups: Option<Vec<String>>,
is_owner: bool,
claims: HashMap<String, Value>,
conditions: HashMap<String, Vec<String>>,
/// Audit slot of the S3 request this principal was built for, when the request
/// is being audited. The principal is the only request-derived value that reaches
/// the managed-SSE code paths, so it doubles as the carrier for the slot.
pub(crate) request_audit: Option<Arc<KmsRequestAudit>>,
/// Test-only decision overrides. Carried per principal rather than in a global slot so
/// concurrent tests cannot observe each other's injection.
#[cfg(any(test, feature = "test-util"))]
pub(crate) test_hooks: Option<TestAuthorizationHooks>,
}
impl SseKmsPrincipal {
/// Build the principal of a request that crossed the S3 authorization boundary.
///
/// Anonymous requests use an empty account, matching the bucket-policy authorization
/// path. This keeps every request that crossed the S3 boundary distinct from trusted
/// internal callers, which are represented by `None`.
pub fn new(
account: String,
groups: Option<Vec<String>>,
is_owner: bool,
claims: HashMap<String, Value>,
conditions: HashMap<String, Vec<String>>,
) -> Self {
Self {
account,
groups,
is_owner,
claims,
conditions,
request_audit: None,
#[cfg(any(test, feature = "test-util"))]
test_hooks: None,
}
}
/// The access key the request authenticated with; empty for an anonymous request.
pub fn account(&self) -> &str {
&self.account
}
pub fn groups(&self) -> &Option<Vec<String>> {
&self.groups
}
pub fn is_owner(&self) -> bool {
self.is_owner
}
pub fn claims(&self) -> &HashMap<String, Value> {
&self.claims
}
/// Policy condition values of the request that do not depend on the resource.
pub fn conditions(&self) -> &HashMap<String, Vec<String>> {
&self.conditions
}
/// Attach the audit slot opened for `request_id`, if that request is being audited
/// (see [`KmsRequestAuditScope::register`]).
#[must_use]
pub fn with_request_id(mut self, request_id: &str) -> Self {
self.request_audit = kms_request_audit(request_id);
self
}
/// Override the enforcement switch and the authorizer for this principal only.
#[cfg(any(test, feature = "test-util"))]
#[must_use]
pub fn with_test_hooks(mut self, hooks: Option<TestAuthorizationHooks>) -> Self {
self.test_hooks = hooks;
self
}
#[cfg(test)]
pub(crate) fn for_test(account: &str, enforced: bool, authorizer: Arc<dyn KmsKeyAuthorizer>) -> Self {
Self {
account: account.to_string(),
groups: None,
is_owner: false,
claims: HashMap::new(),
conditions: HashMap::new(),
request_audit: None,
test_hooks: Some(TestAuthorizationHooks { enforced, authorizer }),
}
}
/// Bind this principal to `audit` so managed-SSE operations performed for it are
/// summarised onto the request's S3 audit entry.
#[cfg(test)]
pub(crate) fn with_request_audit(mut self, audit: Arc<KmsRequestAudit>) -> Self {
self.request_audit = Some(audit);
self
}
}
/// Per-principal override of SSE-KMS key authorization, for tests.
#[cfg(any(test, feature = "test-util"))]
#[derive(Clone)]
pub struct TestAuthorizationHooks {
pub enforced: bool,
pub authorizer: Arc<dyn KmsKeyAuthorizer>,
}
#[cfg(any(test, feature = "test-util"))]
impl std::fmt::Debug for TestAuthorizationHooks {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("TestAuthorizationHooks")
.field("enforced", &self.enforced)
.finish_non_exhaustive()
}
}
-144
View File
@@ -1,144 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Re-wrap a stored data key onto its master key's current version.
use crate::core::keying::context::{build_object_encryption_context, decode_minio_kms_context};
use crate::core::keying::metadata::recode_minio_kms_context;
use crate::error::SseError;
use crate::host::SseHost;
use crate::kms::provider::kms_sse_dek_provider;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_kms::is_data_key_envelope;
use rustfs_utils::http::headers::AMZ_ENCRYPTION_KMS;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER, SSEType,
contains_managed_encryption_metadata, normalize_managed_metadata,
};
use std::collections::HashMap;
/// Outcome of a single object's DEK rewrap attempt.
// Consumed by the bulk rekey sweep in the follow-up PR; tests exercise it now.
#[allow(dead_code)]
#[derive(Debug)]
pub enum ObjectDekRewrapOutcome {
/// The object carries no KMS-wrapped RustFS data-key envelope this build
/// can rewrap: plaintext, SSE-C, or a MinIO-sealed data key.
NotApplicable,
/// The envelope is already on the current master key version and format;
/// the caller must persist nothing, so a sweep re-run converges.
AlreadyCurrent,
/// The envelope was rewrapped. `metadata` holds the overrides to merge
/// into the object's user-defined metadata — every stored copy of the old
/// envelope, replaced — via `ObjectLayer::put_object_metadata`.
Rewrapped { metadata: HashMap<String, String> },
}
/// Rewrap the KMS-wrapped data key in an object's stored metadata onto its
/// master key's current version (and current envelope format), without
/// touching the object's data or its plaintext DEK.
///
/// Mirrors the managed decrypt path byte for byte where it matters: the
/// envelope is located through the same normalized-header resolution, and the
/// encryption context is rebuilt with the same helpers, so an envelope the
/// read path can open is exactly an envelope this can rewrap. The KMS backend
/// owns the format and the no-op decision.
///
/// The returned overrides replace every stored copy of the old envelope
/// (RustFS's internal header and the MinIO-compatible slots that the writer
/// fills with the same bytes). A copy left behind would win a read-path
/// fallback and resurrect the old wrapping, so finding no replaceable copy is
/// an error, never a silent success.
pub async fn rewrap_object_encryption_metadata(
host: &dyn SseHost,
bucket: &str,
key: &str,
metadata: &HashMap<String, String>,
) -> Result<ObjectDekRewrapOutcome, SseError> {
if !contains_managed_encryption_metadata(metadata) {
return Ok(ObjectDekRewrapOutcome::NotApplicable);
}
let encryption_type = match metadata.get("x-amz-server-side-encryption").map(String::as_str) {
Some(AMZ_ENCRYPTION_KMS) => SSEType::SseKms,
Some(_) => SSEType::SseS3,
// MinIO-written objects synthesize the public header on read; their
// sealed data keys are not RustFS envelopes and fall out below.
None => SSEType::SseS3,
};
let normalized_metadata = normalize_managed_metadata(metadata, Some(recode_minio_kms_context));
let Some(envelope_b64) = normalized_metadata
.get(INTERNAL_ENCRYPTION_KEY_HEADER)
.or_else(|| metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER))
else {
return Ok(ObjectDekRewrapOutcome::NotApplicable);
};
let encrypted_data_key = BASE64_STANDARD
.decode_to_vec(envelope_b64)
.map_err(|e| SseError::internal(format!("Failed to decode encrypted key: {e}")))?;
// Only RustFS envelopes are rewrappable here; MinIO's builtin-KMS
// ciphertext is opaque bytes owned by a different root of trust.
if !is_data_key_envelope(&encrypted_data_key) {
return Ok(ObjectDekRewrapOutcome::NotApplicable);
}
let kms_context = if matches!(encryption_type, SSEType::SseKms) {
decode_minio_kms_context(metadata)?
} else {
None
};
let object_context = build_object_encryption_context(bucket, key, kms_context.as_ref());
// The same provider selection as the managed decrypt path.
let provider = kms_sse_dek_provider(host).await?;
let response = provider.rewrap_sse_dek(&encrypted_data_key, &object_context).await?;
if !response.rewrapped {
return Ok(ObjectDekRewrapOutcome::AlreadyCurrent);
}
// Replace every stored copy of the old envelope, keyed by value and
// matched case-insensitively: metadata key casing drifts through the
// storage layer, and an override inserted under a differently-cased name
// would sit beside the old copy instead of replacing it. The MinIO
// sealed-key slots can instead hold a sealed *object* key (rio-v2 writer);
// those bytes differ from the envelope and are untouched — the data key
// they seal is unchanged by a rewrap.
const REWRAP_ENVELOPE_HEADERS: [&str; 4] = [
INTERNAL_ENCRYPTION_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
];
let old_envelope_b64 = envelope_b64.clone();
let new_envelope_b64 = BASE64_STANDARD.encode_to_string(&response.ciphertext);
let mut overrides = HashMap::new();
for (stored_name, stored_value) in metadata {
let is_envelope_slot = REWRAP_ENVELOPE_HEADERS
.iter()
.any(|header| stored_name.eq_ignore_ascii_case(header));
if is_envelope_slot && *stored_value == old_envelope_b64 {
overrides.insert(stored_name.clone(), new_envelope_b64.clone());
}
}
if overrides.is_empty() {
return Err(SseError::internal(
"rewrapped a data-key envelope but found no stored metadata copy to replace; refusing a write that would \
leave the old wrapping live",
));
}
Ok(ObjectDekRewrapOutcome::Rewrapped { metadata: overrides })
}
-177
View File
@@ -1,177 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-key KMS authorization on the SSE-KMS data path.
use crate::core::audit::{record_managed_kms_outcome, stored_envelope_master_key_version};
use crate::core::request::SseKmsPrincipal;
use crate::error::SseError;
use crate::host::SseHost;
use crate::{LOG_COMPONENT_STORAGE, LOG_SUBSYSTEM_SSE};
use rustfs_config::ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY;
use rustfs_policy::policy::action::KmsAction;
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::http::object_encryption_keys::{SSEType, stored_managed_encryption_key};
use std::collections::HashMap;
use tracing::debug;
/// Ask the authorizer that decides for `principal`: the host's, unless a test overrode it.
async fn is_kms_key_allowed(
host: &dyn SseHost,
principal: &SseKmsPrincipal,
action: KmsAction,
key_id: &str,
) -> Result<bool, SseError> {
#[cfg(any(test, feature = "test-util"))]
if let Some(hooks) = principal.test_hooks.as_ref() {
return hooks.authorizer.is_allowed(principal, action, key_id).await;
}
host.kms_key_authorizer().is_allowed(principal, action, key_id).await
}
/// Whether per-key KMS authorization is enforced on the SSE-KMS data path.
fn sse_kms_key_policy_enforced(principal: Option<&SseKmsPrincipal>) -> bool {
#[cfg(any(test, feature = "test-util"))]
if let Some(hooks) = principal.and_then(|candidate| candidate.test_hooks.as_ref()) {
return hooks.enforced;
}
#[cfg(not(any(test, feature = "test-util")))]
let _ = principal;
crate::startup::key_policy_enforced()
}
/// The principal whose KMS key permissions this operation must satisfy, if any.
///
/// Scoping is limited to SSE-KMS, matching AWS: SSE-S3 wraps its data key with a
/// server-owned key the caller never names, and SSE-C never reaches KMS at all.
pub(crate) fn kms_authorization_subject(
enforced: bool,
principal: Option<&SseKmsPrincipal>,
sse_type: SSEType,
) -> Option<&SseKmsPrincipal> {
if !enforced || !matches!(sse_type, SSEType::SseKms) {
return None;
}
principal
}
/// Authorize `action` against `key_id` for the caller behind this SSE operation.
///
/// Runs before the key is used, so a denied request cannot distinguish an unauthorized
/// key from a disabled or pending-deletion one.
pub(crate) async fn authorize_sse_kms_key(
host: &dyn SseHost,
principal: Option<&SseKmsPrincipal>,
sse_type: SSEType,
action: KmsAction,
key_id: &str,
) -> Result<(), SseError> {
let Some(principal) = kms_authorization_subject(sse_kms_key_policy_enforced(principal), principal, sse_type) else {
return Ok(());
};
if is_kms_key_allowed(host, principal, action, key_id).await? {
return Ok(());
}
debug!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "sse_kms_key_authorization_denied",
account = %principal.account,
action = ?action,
"Principal is not authorized for the KMS key resolved for this request"
);
// One warn per process, not per request: anonymous denials are driven by
// unauthenticated traffic, so a per-request warn would let anyone flood the
// log. Per-request detail stays on the audit entry and the debug event above.
if principal.account.is_empty() {
static ANONYMOUS_DENIAL_WARNED: std::sync::Once = std::sync::Once::new();
ANONYMOUS_DENIAL_WARNED.call_once(|| {
tracing::warn!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "sse_kms_anonymous_key_authorization_denied",
action = ?action,
"Anonymous requests are being denied by SSE-KMS per-key authorization: anonymous \
callers hold no kms grants, so a public bucket serving SSE-KMS objects is \
incompatible with {ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true. Reported once per \
process; per-request denials are on audit entries and at debug level."
);
});
}
Err(SseError {
code: S3ErrorCode::AccessDenied,
message: "Access Denied".to_string(),
source: None,
})
}
/// Authorize `kms:Decrypt` for an object whose stored encryption material is unwrapped
/// outside [`sse_decryption`].
///
/// The copy-source read resolves its material inside the object layer, which has no
/// request identity, so the S3 layer has to run the check itself.
pub async fn authorize_sse_kms_object_read(
host: &dyn SseHost,
principal: Option<&SseKmsPrincipal>,
metadata: &HashMap<String, String>,
) -> Result<(), SseError> {
let Some((sse_type, key_id)) = stored_managed_encryption_key(metadata) else {
return Ok(());
};
let result = authorize_sse_kms_key(host, principal, sse_type, KmsAction::DecryptAction, &key_id).await;
// Only a denial is recorded here: an allowed read goes on to unwrap the key,
// and that operation reports its own outcome.
if let Err(error) = &result {
record_managed_kms_outcome(
principal,
sse_type,
Some(&key_id),
|| stored_envelope_master_key_version(metadata),
Err(error),
);
}
result
}
/// Authorize the current request to write under the upload's stored KMS key.
/// Upload IDs do not confer the creator's KMS permissions on another caller.
pub async fn authorize_sse_kms_multipart_write(
host: &dyn SseHost,
principal: Option<&SseKmsPrincipal>,
metadata: &HashMap<String, String>,
) -> Result<(), SseError> {
let Some((sse_type, key_id)) = stored_managed_encryption_key(metadata) else {
return Ok(());
};
let result = authorize_sse_kms_key(host, principal, sse_type, KmsAction::GenerateDataKeyAction, &key_id).await;
if let Err(error) = &result {
record_managed_kms_outcome(
principal,
sse_type,
Some(&key_id),
|| stored_envelope_master_key_version(metadata),
Err(error),
);
}
result
}
-454
View File
@@ -1,454 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! SSE request headers in, SSE response headers out.
use crate::core::keying::metadata::{normalize_encryption_metadata_case, recode_minio_kms_context, stored_ssec_metadata};
use crate::core::s3::ssec::ssec_invalid_request;
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use http::{HeaderMap, HeaderName, HeaderValue};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::http::headers::{
AMZ_ENCRYPTION_AES, AMZ_ENCRYPTION_KMS, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT,
SSEC_ALGORITHM_HEADER, SSEC_KEY_MD5_HEADER,
};
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_CONTEXT_HEADER, INTERNAL_ENCRYPTION_KEY_ID_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER, SSEType,
normalize_managed_metadata,
};
use std::collections::HashMap;
// The same header names s3s publishes. Owned so this module does not import s3s.
pub(crate) const X_AMZ_SERVER_SIDE_ENCRYPTION: HeaderName = HeaderName::from_static("x-amz-server-side-encryption");
pub(crate) const X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID: HeaderName =
HeaderName::from_static("x-amz-server-side-encryption-aws-kms-key-id");
pub(crate) const X_AMZ_SERVER_SIDE_ENCRYPTION_CONTEXT: HeaderName =
HeaderName::from_static("x-amz-server-side-encryption-context");
pub(crate) const X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM: HeaderName =
HeaderName::from_static("x-amz-server-side-encryption-customer-algorithm");
pub(crate) const X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5: HeaderName =
HeaderName::from_static("x-amz-server-side-encryption-customer-key-md5");
#[inline]
pub(crate) fn sse_invalid_argument(message: &str) -> SseError {
SseError {
code: S3ErrorCode::InvalidArgument,
message: message.to_string(),
source: None,
}
}
/// SSE-C parameters extracted from headers (algorithm, key, key MD5).
pub type SsecParamsFromHeaders = (Option<String>, Option<String>, Option<String>);
/// Extract SSE-C parameters from request headers.
/// Used as fallback when the S3 layer does not populate them in the input struct.
///
/// Returns an error if an SSE-C header is present but cannot be parsed as valid UTF-8,
/// ensuring malformed headers do not bypass validation.
pub fn extract_ssec_params_from_headers(headers: &HeaderMap) -> Result<SsecParamsFromHeaders, SseError> {
let algorithm = match headers.get("x-amz-server-side-encryption-customer-algorithm") {
None => Ok(None),
Some(v) => v
.to_str()
.map(|s| Some(s.to_string()))
.map_err(|_| sse_invalid_argument("The x-amz-server-side-encryption-customer-algorithm header must be valid UTF-8.")),
}?;
let key = match headers.get("x-amz-server-side-encryption-customer-key") {
None => Ok(None),
Some(v) => v
.to_str()
.map(|s| Some(s.to_string()))
.map_err(|_| sse_invalid_argument("The x-amz-server-side-encryption-customer-key header must be valid UTF-8.")),
}?;
let key_md5 = match headers.get("x-amz-server-side-encryption-customer-key-md5") {
None => Ok(None),
Some(v) => v
.to_str()
.map(|s| Some(s.to_string()))
.map_err(|_| sse_invalid_argument("The x-amz-server-side-encryption-customer-key-md5 header must be valid UTF-8.")),
}?;
Ok((algorithm, key, key_md5))
}
/// Extract x-amz-server-side-encryption from request headers.
/// Used as fallback when the S3 layer does not populate it in the input struct.
///
/// Returns an error if the header is present but cannot be parsed as valid UTF-8,
/// ensuring malformed headers do not bypass validation.
pub fn extract_server_side_encryption_from_headers(headers: &HeaderMap) -> Result<Option<String>, SseError> {
match headers.get("x-amz-server-side-encryption") {
None => Ok(None),
Some(v) => v
.to_str()
.map(|s| Some(s.to_string()))
.map_err(|_| sse_invalid_argument("The x-amz-server-side-encryption header must be valid UTF-8.")),
}
}
pub fn extract_ssekms_context_from_headers(headers: &HeaderMap) -> Result<Option<HashMap<String, String>>, SseError> {
let Some(v) = headers.get(AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT) else {
return Ok(None);
};
let value = v
.to_str()
.map_err(|_| sse_invalid_argument("The x-amz-server-side-encryption-context header must be valid UTF-8."))?;
let decoded = BASE64_STANDARD.decode_to_vec(value).map_err(|_| {
sse_invalid_argument("The x-amz-server-side-encryption-context header must be valid base64-encoded JSON.")
})?;
serde_json::from_slice(&decoded).map(Some).map_err(|_| {
sse_invalid_argument("The x-amz-server-side-encryption-context header must be a base64-encoded JSON object.")
})
}
/// Rejects a client encryption context that would replace the location binding.
///
/// The request context binds the data key to `{bucket: bucket/key}` only when the
/// client supplied no entry under that name, and only the client context is
/// persisted. A client entry keyed by the bucket name therefore travels with the
/// object and replaces the binding on every read, so the envelope would no longer
/// be tied to the object's location. Only the write side rejects it: objects stored
/// with such an entry stay readable because the read side rebuilds the context
/// unchanged.
pub(crate) fn reject_kms_context_location_override(
bucket: &str,
ssekms_context: Option<&HashMap<String, String>>,
) -> Result<(), SseError> {
if ssekms_context.is_some_and(|context| context.contains_key(bucket)) {
return Err(sse_invalid_argument(&format!(
"The x-amz-server-side-encryption-context header contains the key \"{bucket}\", which equals the bucket name; \
that key is reserved for binding the data key to the object's location."
)));
}
Ok(())
}
#[inline]
pub fn validate_sse_headers_for_write(
server_side_encryption: Option<&str>,
ssekms_key_id: Option<&String>,
ssekms_context: Option<&HashMap<String, String>>,
sse_customer_algorithm: Option<&String>,
sse_customer_key: Option<&String>,
sse_customer_key_md5: Option<&String>,
require_sse_customer_key: bool,
) -> Result<(), SseError> {
if let Some(sse) = server_side_encryption {
let s = sse;
if s != AMZ_ENCRYPTION_AES && s != AMZ_ENCRYPTION_KMS {
return Err(sse_invalid_argument(
"The SSE algorithm specified is not supported. The valid values are AES256 or aws:kms.",
));
}
}
let has_ssec_headers = sse_customer_algorithm.is_some() || sse_customer_key.is_some() || sse_customer_key_md5.is_some();
let has_managed_headers = server_side_encryption.is_some() || ssekms_key_id.is_some() || ssekms_context.is_some();
if has_ssec_headers {
if has_managed_headers {
return Err(sse_invalid_argument(
"The SSE-C and managed server-side encryption headers cannot be used together.",
));
}
let has_valid_ssec_headers = if require_sse_customer_key {
matches!(
(sse_customer_algorithm, sse_customer_key, sse_customer_key_md5),
(Some(_), Some(_), Some(_))
)
} else {
matches!((sse_customer_algorithm, sse_customer_key_md5), (Some(_), Some(_)))
};
if !has_valid_ssec_headers {
let message = if require_sse_customer_key {
"Missing SSE-C parameters. Algorithm, customer key and customer key MD5 are all required."
} else {
"Missing SSE-C parameters. Algorithm and customer key MD5 are required."
};
return Err(ssec_invalid_request(message));
}
}
if ssekms_key_id.is_some() && server_side_encryption.is_none_or(|sse| sse != AMZ_ENCRYPTION_KMS) {
return Err(sse_invalid_argument(
"The SSE-KMS key ID header can only be used when x-amz-server-side-encryption is set to aws:kms.",
));
}
if ssekms_context.is_some() && server_side_encryption.is_none_or(|sse| sse != AMZ_ENCRYPTION_KMS) {
return Err(sse_invalid_argument(
"The SSE-KMS context header can only be used when x-amz-server-side-encryption is set to aws:kms.",
));
}
Ok(())
}
#[inline]
pub fn validate_sse_headers_for_read(metadata: &HashMap<String, String>, headers: &HeaderMap) -> Result<(), SseError> {
let has_req_ssec = headers.contains_key("x-amz-server-side-encryption-customer-algorithm")
|| headers.contains_key("x-amz-server-side-encryption-customer-key")
|| headers.contains_key("x-amz-server-side-encryption-customer-key-md5");
let has_req_sse = headers.contains_key("x-amz-server-side-encryption")
|| headers.contains_key("x-amz-server-side-encryption-aws-kms-key-id")
|| headers.contains_key("x-amz-server-side-encryption-context");
let is_object_ssec = stored_ssec_metadata(metadata);
let is_object_sse = metadata.contains_key("x-amz-server-side-encryption");
if is_object_ssec {
if has_req_sse {
return Err(sse_invalid_argument(
"Server-side encryption headers cannot be used with an object encrypted using SSE-C.",
));
}
return Ok(());
}
if is_object_sse && has_req_ssec {
return Err(sse_invalid_argument(
"SSE-C headers cannot be used with an object encrypted using server-side managed encryption.",
));
}
if has_req_ssec {
return Err(ssec_invalid_request(
"The object was stored without SSE-C. The correct SSE-C parameters must not be provided.",
));
}
if has_req_sse {
return Err(sse_invalid_argument(
"The object is not encrypted with server-side encryption. Do not provide server-side encryption headers.",
));
}
Ok(())
}
/// The public `x-amz-server-side-encryption` value a managed scheme reports.
pub(crate) fn managed_sse_public_header(sse_type: SSEType) -> &'static str {
match sse_type {
SSEType::SseKms => AMZ_ENCRYPTION_KMS,
// SSE-C never reaches the managed path; reporting AES256 keeps this
// total without inventing a third public value.
SSEType::SseS3 | SSEType::SseC => AMZ_ENCRYPTION_AES,
}
}
pub fn build_ssec_read_headers(algorithm: Option<&String>, key: Option<&String>, key_md5: Option<&String>) -> HeaderMap {
let mut headers = HeaderMap::new();
if let Some(algorithm) = algorithm
&& let Ok(mut value) = HeaderValue::from_str(algorithm.as_str())
{
value.set_sensitive(true);
headers.insert(AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, value);
}
if let Some(key) = key
&& let Ok(mut value) = HeaderValue::from_str(key.as_str())
{
value.set_sensitive(true);
headers.insert(AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, value);
}
if let Some(key_md5) = key_md5
&& let Ok(mut value) = HeaderValue::from_str(key_md5.as_str())
{
value.set_sensitive(true);
headers.insert(AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, value);
}
headers
}
/// Response headers derived for a decrypted read without unwrapping the data key.
#[derive(Debug)]
pub struct SseReadResponseHeaders {
pub server_side_encryption: String,
pub sse_customer_algorithm: Option<String>,
pub sse_customer_key_md5: Option<String>,
pub ssekms_key_id: Option<String>,
}
fn invalid_sse_response_metadata() -> SseError {
SseError {
code: S3ErrorCode::InternalError,
message: "Persisted object encryption metadata is invalid.".to_string(),
source: None,
}
}
fn insert_sse_response_header(headers: &mut HeaderMap, name: HeaderName, value: &str) -> Result<(), SseError> {
let value = HeaderValue::from_str(value).map_err(|_| invalid_sse_response_metadata())?;
headers.insert(name, value);
Ok(())
}
/// Project response headers after the caller has completed its existing read
/// validation and KMS authorization path.
pub fn project_sse_read_response_headers(
metadata: &HashMap<String, String>,
sse_customer_algorithm: Option<&String>,
sse_customer_key_md5: Option<&String>,
) -> Result<HeaderMap, SseError> {
if metadata.keys().any(|key| {
key.eq_ignore_ascii_case(AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY)
|| key.eq_ignore_ascii_case(AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT)
}) {
return Err(invalid_sse_response_metadata());
}
let metadata = normalize_encryption_metadata_case(metadata).map_err(|_| invalid_sse_response_metadata())?;
let public_mode = metadata.get("x-amz-server-side-encryption").map(String::as_str);
let customer_algorithm = metadata.get(SSEC_ALGORITHM_HEADER).map(String::as_str);
let public_mode = match public_mode {
Some(AMZ_ENCRYPTION_AES) => Some(SSEType::SseS3),
Some(AMZ_ENCRYPTION_KMS) => Some(SSEType::SseKms),
Some(_) => return Err(invalid_sse_response_metadata()),
None => None,
};
if customer_algorithm.is_some_and(|algorithm| algorithm != AMZ_ENCRYPTION_AES) {
return Err(invalid_sse_response_metadata());
}
let mode = if customer_algorithm.is_some() {
if public_mode == Some(SSEType::SseKms) {
return Err(invalid_sse_response_metadata());
}
Some(SSEType::SseC)
} else {
public_mode
};
for marker_mode in [
metadata
.contains_key(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER)
.then_some(SSEType::SseC),
metadata
.contains_key(MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER)
.then_some(SSEType::SseS3),
metadata
.contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_SEALED_KEY_HEADER)
.then_some(SSEType::SseKms),
]
.into_iter()
.flatten()
{
if mode != Some(marker_mode) {
return Err(invalid_sse_response_metadata());
}
}
let Some(mode) = mode else {
if metadata
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
{
return Err(invalid_sse_response_metadata());
}
return Ok(HeaderMap::new());
};
let mut key_id = None;
for value in [
metadata.get("x-amz-server-side-encryption-aws-kms-key-id"),
metadata.get(INTERNAL_ENCRYPTION_KEY_ID_HEADER),
metadata.get(MINIO_INTERNAL_ENCRYPTION_KMS_KEY_ID_HEADER),
]
.into_iter()
.flatten()
{
if key_id.is_some_and(|current| current != value) {
return Err(invalid_sse_response_metadata());
}
key_id = Some(value);
}
let normalized_managed = normalize_managed_metadata(metadata.as_ref(), Some(recode_minio_kms_context));
let mut response_headers = HeaderMap::with_capacity(3);
match mode {
SSEType::SseS3 => {
if metadata.contains_key("x-amz-server-side-encryption-aws-kms-key-id") || metadata.contains_key(SSEC_KEY_MD5_HEADER)
{
return Err(invalid_sse_response_metadata());
}
response_headers.insert(X_AMZ_SERVER_SIDE_ENCRYPTION, HeaderValue::from_static(AMZ_ENCRYPTION_AES));
}
SSEType::SseKms => {
if metadata.contains_key(SSEC_KEY_MD5_HEADER) {
return Err(invalid_sse_response_metadata());
}
let key_id = key_id
.map(String::as_str)
.filter(|value| !value.is_empty())
.ok_or_else(invalid_sse_response_metadata)?;
response_headers.insert(X_AMZ_SERVER_SIDE_ENCRYPTION, HeaderValue::from_static(AMZ_ENCRYPTION_KMS));
insert_sse_response_header(&mut response_headers, X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID, key_id)?;
if metadata.contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER)
&& !normalized_managed.contains_key(INTERNAL_ENCRYPTION_CONTEXT_HEADER)
{
return Err(invalid_sse_response_metadata());
}
if let Some(context) = normalized_managed.get(INTERNAL_ENCRYPTION_CONTEXT_HEADER) {
serde_json::from_str::<HashMap<String, String>>(context).map_err(|_| invalid_sse_response_metadata())?;
let encoded = BASE64_STANDARD.encode_to_string(context);
insert_sse_response_header(&mut response_headers, X_AMZ_SERVER_SIDE_ENCRYPTION_CONTEXT, &encoded)?;
}
}
SSEType::SseC => {
if key_id.is_some() || normalized_managed.contains_key(INTERNAL_ENCRYPTION_CONTEXT_HEADER) {
return Err(invalid_sse_response_metadata());
}
let algorithm = sse_customer_algorithm
.map(|value| value.as_ref())
.filter(|value| *value == AMZ_ENCRYPTION_AES)
.ok_or_else(|| {
ssec_invalid_request(
"The object was stored using a form of Server Side Encryption. The correct parameters must be provided to retrieve the object.",
)
})?;
let key_md5 = sse_customer_key_md5
.map(|value| value.as_ref())
.ok_or_else(|| ssec_invalid_request("Missing SSE-C customer key MD5."))?;
let stored_md5 = metadata.get(SSEC_KEY_MD5_HEADER).ok_or_else(invalid_sse_response_metadata)?;
if stored_md5 != key_md5 {
return Err(ssec_invalid_request(
"The provided encryption parameters did not match the ones used originally to encrypt the object.",
));
}
insert_sse_response_header(&mut response_headers, X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, algorithm)?;
insert_sse_response_header(&mut response_headers, X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, key_md5)?;
}
}
Ok(response_headers)
}
-20
View File
@@ -1,20 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The schemes S3 defines: SSE-S3, SSE-KMS and SSE-C, as a request names them and a response reports them.
pub(crate) mod authorization;
pub(crate) mod headers;
pub(crate) mod scheme;
pub(crate) mod ssec;
-288
View File
@@ -1,288 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Which SSE scheme and which key a request resolves to: the request first, then the bucket default.
use crate::core::s3::headers::validate_sse_headers_for_write;
use crate::error::SseError;
use crate::host::SseHost;
use crate::{LOG_COMPONENT_STORAGE, LOG_SUBSYSTEM_SSE};
use rustfs_utils::http::headers::{AMZ_ENCRYPTION_AES, AMZ_ENCRYPTION_KMS};
use rustfs_utils::http::object_encryption_keys::SSEType;
use std::collections::HashMap;
use tracing::debug;
pub const DEFAULT_SSE_ALGORITHM: &str = "AES256";
pub(crate) const SUPPORT_SSE_ALGORITHMS: &[&str] = &[DEFAULT_SSE_ALGORITHM];
/// SSE configuration resolved from request and bucket defaults
#[derive(Debug)]
pub(crate) struct SseConfiguration {
/// Effective server-side encryption algorithm (after considering bucket defaults)
pub effective_sse: String,
/// Effective KMS key ID (after considering bucket defaults)
pub effective_kms_key_id: Option<String>,
}
/// Managed SSE resolved from a bucket default encryption rule on a write path.
///
/// The single mapping shared by every writer: the resolver in this crate, and
/// the server's PUT, COPY and extract paths. Unknown algorithms fall back to AES256 rather
/// than to `None`. Resolving `None` instead lets a same-name copy under a
/// malformed bucket default pass the `copy_changes_encryption` guard and take
/// the metadata-only shortcut while this layer still encrypts: fresh DEK
/// metadata is committed beside the untouched plaintext blocks and the object
/// becomes unreadable. PutBucketEncryption refuses unknown algorithms, so this
/// is reachable only through a configuration stored before that check or
/// through hand-edited bucket metadata (backlog#1826).
pub fn bucket_default_write_sse(sse_algorithm: &str) -> String {
match sse_algorithm {
"AES256" => String::from(AMZ_ENCRYPTION_AES),
"aws:kms" => String::from(AMZ_ENCRYPTION_KMS),
_ => String::from(AMZ_ENCRYPTION_AES),
}
}
/// Prepare SSE configuration by resolving request parameters with bucket defaults
///
/// This function:
/// 1. Queries bucket default encryption configuration
/// 2. Resolves effective encryption (request overrides bucket default)
/// 3. Prepares metadata headers for managed SSE
///
/// # Arguments
/// * `bucket` - Bucket name
/// * `server_side_encryption` - SSE algorithm from request (SSE-S3 or SSE-KMS)
/// * `ssekms_key_id` - KMS key ID from request
/// * `sse_customer_algorithm` - SSE-C algorithm from request
///
/// # Returns
/// `SseConfiguration` with resolved encryption parameters and metadata headers
pub(crate) async fn prepare_sse_configuration(
host: &dyn SseHost,
bucket: &str,
server_side_encryption: Option<String>,
ssekms_key_id: Option<String>,
) -> Result<Option<SseConfiguration>, SseError> {
if let Some(server_side_encryption) = server_side_encryption.clone()
&& server_side_encryption.as_str() == AMZ_ENCRYPTION_AES
{
return Ok(Some(SseConfiguration {
effective_sse: server_side_encryption,
effective_kms_key_id: None,
}));
}
if let Some(server_side_encryption) = server_side_encryption.clone()
&& let Some(ssekms_key_id) = ssekms_key_id
{
return Ok(Some(SseConfiguration {
effective_sse: server_side_encryption,
effective_kms_key_id: Some(ssekms_key_id),
}));
}
// Get bucket default encryption configuration.
let bucket_encryption = host.bucket_encryption(bucket).await;
debug!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "bucket_sse_config_lookup",
bucket = %bucket,
found = matches!(bucket_encryption, Ok(Some(_))),
"Bucket SSE configuration lookup completed"
);
let Some(bucket_encryption) = bucket_encryption? else {
// The bucket has no SSE config. If the user explicitly requested
// aws:kms, we must honor that — return the explicit SSE header so
// downstream logic can try (and fail if KMS is unavailable).
return Ok(server_side_encryption.map(|sse| SseConfiguration {
effective_sse: sse,
effective_kms_key_id: ssekms_key_id,
}));
};
let default_rule = bucket_encryption.default_rule.as_ref();
let effective_sse = server_side_encryption.clone().or_else(|| {
default_rule.map(|sse| {
debug!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "bucket_sse_default_applied",
bucket = %bucket,
algorithm = sse.sse_algorithm.as_str(),
has_kms_key_id = sse.kms_master_key_id.is_some(),
"Bucket SSE default resolved"
);
bucket_default_write_sse(&sse.sse_algorithm)
})
});
let Some(effective_sse) = effective_sse else {
return Ok(None);
};
debug!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
event = "effective_sse_resolved",
bucket = %bucket,
requested = ?server_side_encryption,
effective = ?effective_sse,
"Resolved effective SSE configuration"
);
let effective_kms_key_id = resolve_effective_kms_key_id(Some(&effective_sse), ssekms_key_id, || {
default_rule.and_then(|sse| sse.kms_master_key_id.clone())
});
Ok(Some(SseConfiguration {
effective_sse,
effective_kms_key_id,
}))
}
pub(crate) fn resolve_effective_kms_key_id<F>(
effective_sse: Option<&str>,
requested_kms_key_id: Option<String>,
bucket_default_kms_key_id: F,
) -> Option<String>
where
F: FnOnce() -> Option<String>,
{
if effective_sse.is_none_or(|sse| sse != AMZ_ENCRYPTION_KMS) {
return requested_kms_key_id;
}
requested_kms_key_id.or_else(bucket_default_kms_key_id)
}
/// The SSE headers of a write request, as the S3 layer received them.
pub(crate) struct WriteSchemeHeaders<'a> {
pub(crate) server_side_encryption: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) ssekms_context: Option<&'a HashMap<String, String>>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
}
/// The scheme a write resolved to, in S3's terms.
#[derive(Clone)]
pub(crate) enum WriteScheme {
/// `AES256`: the server chooses the wrapping key and the caller names none.
SseS3,
/// `aws:kms`, under the key the request or the bucket default named, if either did.
SseKms { key_id: Option<String> },
/// A key the client supplies; only its MD5 is ever stored.
SseC {
algorithm: String,
key: String,
key_md5: String,
},
}
impl std::fmt::Debug for WriteScheme {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(match self {
Self::SseS3 => "SseS3",
Self::SseKms { .. } => "SseKms(..)",
Self::SseC { .. } => "SseC(..)",
})
}
}
/// Decide the scheme of a write: validate the headers, then the request first and the
/// bucket default second.
///
/// Every write goes through here, an object and a multipart session alike, so the two
/// cannot resolve the same headers differently.
pub(crate) async fn resolve_write_scheme(
host: &dyn SseHost,
bucket: &str,
headers: WriteSchemeHeaders<'_>,
) -> Result<Option<WriteScheme>, SseError> {
validate_sse_headers_for_write(
headers.server_side_encryption.as_deref(),
headers.ssekms_key_id.as_ref(),
headers.ssekms_context,
headers.sse_customer_algorithm.as_ref(),
headers.sse_customer_key.as_ref(),
headers.sse_customer_key_md5.as_ref(),
true,
)?;
// Validation admits SSE-C headers only as a complete set, and never beside the
// managed ones, so their presence alone decides.
if let (Some(algorithm), Some(key), Some(key_md5)) =
(headers.sse_customer_algorithm, headers.sse_customer_key, headers.sse_customer_key_md5)
{
return Ok(Some(WriteScheme::SseC { algorithm, key, key_md5 }));
}
let Some(config) = prepare_sse_configuration(host, bucket, headers.server_side_encryption, headers.ssekms_key_id).await?
else {
return Ok(None);
};
Ok(match config.effective_sse.as_str() {
AMZ_ENCRYPTION_AES => {
// Validation admits a key id only beside `aws:kms`, and the bucket default's
// key is consulted only for `aws:kms`.
debug_assert!(config.effective_kms_key_id.is_none(), "SSE-S3 never names a KMS key");
Some(WriteScheme::SseS3)
}
AMZ_ENCRYPTION_KMS => Some(WriteScheme::SseKms {
key_id: config.effective_kms_key_id,
}),
_ => None,
})
}
/// Scheme a managed-SSE header names, defaulting to SSE-S3 the same way the
/// encryption and decryption paths do.
pub(crate) fn managed_sse_type(server_side_encryption: &str) -> SSEType {
match server_side_encryption {
AMZ_ENCRYPTION_KMS => SSEType::SseKms,
_ => SSEType::SseS3,
}
}
/// Key id an SSE-KMS read authorizes and reports when the stored metadata
/// carries none (legacy envelopes and MinIO-written objects predate the
/// explicit key-id headers).
///
/// The KMS service's configured default key is the key such envelopes were
/// actually wrapped under, so per-key authorization must target it — the bare
/// literal `default` would silently escape any policy narrowed to real key
/// ids. The literal remains only as the last resort when no service is
/// running, matching what the legacy write path recorded implicitly.
pub(crate) async fn resolve_stored_kms_key_id(host: &dyn SseHost, stored: Option<String>) -> String {
if let Some(key_id) = stored {
return key_id;
}
if let Some(service) = host.encryption_service().await
&& let Some(key_id) = service.get_default_key_id()
{
return key_id.clone();
}
"default".to_string()
}
/// Check if the server_side_encryption is a managed SSE type (SSE-S3 or SSE-KMS)
#[inline]
pub(crate) fn is_managed_sse(server_side_encryption: &str) -> bool {
matches!(server_side_encryption, "AES256" | "aws:kms")
}
-217
View File
@@ -1,217 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! SSE-C: validating the customer-provided key and matching it against the stored object.
use crate::core::keying::metadata::{minio_sealed_ssec_metadata, stored_ssec_metadata};
use crate::core::s3::scheme::{DEFAULT_SSE_ALGORITHM, SUPPORT_SSE_ALGORITHMS};
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use md5::{Digest as Md5Digest, Md5};
use rustfs_s3_types::S3ErrorCode;
use std::collections::HashMap;
use tracing::error;
fn md5_bytes(input: impl AsRef<[u8]>) -> [u8; 16] {
let mut hasher = Md5::new();
hasher.update(input.as_ref());
hasher.finalize().into()
}
pub(crate) fn md5_base64(input: impl AsRef<[u8]>) -> String {
BASE64_STANDARD.encode_to_string(md5_bytes(input))
}
/// Validated SSE-C parameters
#[derive(Debug, Clone)]
pub(crate) struct ValidatedSsecParams {
/// Encryption algorithm (always "AES256" for SSE-C)
pub algorithm: String,
/// Decoded encryption key bytes (32 bytes for AES-256)
pub key_bytes: [u8; 32],
/// Base64-encoded MD5 of the key
pub key_md5: String,
}
/// SSE-C parameters from client request
#[derive(Debug, Clone)]
pub(crate) struct SsecParams {
/// Encryption algorithm
pub algorithm: String,
/// Base64-encoded encryption key
pub key: String,
/// Base64-encoded MD5 of the key
pub key_md5: String,
}
/// Validate SSE-C parameters from client request
///
/// Validates:
/// 1. Algorithm is "AES256"
/// 2. Key is valid Base64 and exactly 32 bytes
/// 3. MD5 hash matches the key
///
/// # Returns
/// `ValidatedSsecParams` with decoded key bytes
pub(crate) fn validate_ssec_params(params: SsecParams) -> Result<ValidatedSsecParams, SseError> {
if !SUPPORT_SSE_ALGORITHMS.contains(&params.algorithm.as_str()) {
return Err(ssec_invalid_request(&format!(
"Unsupported SSE-C algorithm. Only {DEFAULT_SSE_ALGORITHM} is supported."
)));
}
let key_bytes = BASE64_STANDARD.decode_to_vec(&params.key).map_err(|e| {
error!("Failed to decode SSE-C key: {}", e);
ssec_invalid_request("Invalid SSE-C key: not valid Base64.")
})?;
if key_bytes.len() != 32 {
return Err(ssec_invalid_request(&format!(
"SSE-C key must be 32 bytes (256 bits), got {} bytes.",
key_bytes.len()
)));
}
let computed_md5 = md5_base64(&key_bytes);
if computed_md5 != params.key_md5 {
return Err(ssec_invalid_request(
"The calculated MD5 hash of the key did not match the hash that was provided.",
));
}
let key_array: [u8; 32] = key_bytes
.try_into()
.map_err(|_| ssec_invalid_request("SSE-C key must be exactly 32 bytes."))?;
Ok(ValidatedSsecParams {
algorithm: params.algorithm,
key_bytes: key_array,
key_md5: params.key_md5,
})
}
/// Verify SSE-C key matches the stored metadata.
///
/// Used during GetObject/HeadObject to ensure the client provided the correct key.
/// Returns 400 InvalidRequest on mismatch, consistent with AWS S3 behavior.
pub(crate) fn verify_ssec_key_match(
provided_md5: &str,
stored_md5: Option<&String>,
minio_sealed_ssec: bool,
) -> Result<(), SseError> {
match stored_md5 {
Some(stored) if stored == provided_md5 => Ok(()),
Some(_) => Err(ssec_invalid_request(
"The provided encryption parameters did not match the ones used originally to encrypt the object.",
)),
// MinIO never persists the customer-key MD5, so this early check has
// nothing to compare against on a migrated object. The key itself is
// still proven: a wrong key fails the AEAD unseal of the sealed object
// key. The check exists only for the friendlier early error, so its
// absence is tolerated for exactly that shape and no other.
None if minio_sealed_ssec => Ok(()),
None => Err(ssec_invalid_request("Object has no stored SSE-C key metadata.")),
}
}
/// Algorithm an SSE-C object recorded; objects that recorded none were written as `AES256`.
pub(crate) fn stored_ssec_algorithm(metadata: &HashMap<String, String>) -> &str {
metadata
.get("x-amz-server-side-encryption-customer-algorithm")
.map(|s| s.as_str())
.unwrap_or("AES256")
}
/// The customer key and MD5 of a read of an SSE-C object, once both are present and
/// the MD5 matches what the object recorded.
///
/// The key itself is not validated here; the caller does that next, so a request
/// that names the wrong key learns of the mismatch before anything else.
pub(crate) fn require_matching_ssec_key<'r>(
metadata: &HashMap<String, String>,
sse_customer_key: Option<&'r String>,
sse_customer_key_md5: Option<&'r String>,
) -> Result<(&'r String, &'r String), SseError> {
let (key, key_md5) = match (sse_customer_key, sse_customer_key_md5) {
(Some(k), Some(md5)) => (k, md5),
_ => {
return Err(ssec_invalid_request(
"The object was stored using a form of Server Side Encryption. \
The correct parameters must be provided to retrieve the object.",
));
}
};
// Verify that the provided key MD5 matches the stored MD5 for security
let stored_md5 = metadata.get("x-amz-server-side-encryption-customer-key-md5");
verify_ssec_key_match(key_md5, stored_md5, minio_sealed_ssec_metadata(metadata))?;
Ok((key, key_md5))
}
/// Validate that the SSE-C headers required for reading an SSE-C encrypted object
/// are present in the request. This is used by HeadObject which does not decrypt
/// the data but still must verify the caller holds the correct key.
///
/// Performs full validation: decodes the customer key, recomputes its MD5,
/// verifies the client-provided MD5 header matches the key, then compares
/// the computed MD5 against the stored metadata. This prevents a client from
/// bypassing validation by guessing/obtaining only the stored MD5 without
/// possessing the actual encryption key.
///
/// Returns `Ok(())` if either the object is not SSE-C encrypted, or valid SSE-C
/// headers are provided and the key matches. Returns 400 InvalidRequest otherwise.
pub fn validate_ssec_for_read(
metadata: &HashMap<String, String>,
sse_customer_key: Option<&String>,
sse_customer_key_md5: Option<&String>,
) -> Result<(), SseError> {
if !stored_ssec_metadata(metadata) {
return Ok(());
}
let (key, key_md5) = match (sse_customer_key, sse_customer_key_md5) {
(Some(k), Some(md5)) => (k, md5),
_ => {
return Err(ssec_invalid_request(
"The object was stored using a form of Server Side Encryption. \
The correct parameters must be provided to retrieve the object.",
));
}
};
// Full param validation: decode key, verify 32 bytes, recompute MD5
// from actual key bytes and compare to the client-provided MD5 header.
let algorithm = metadata
.get("x-amz-server-side-encryption-customer-algorithm")
.cloned()
.unwrap_or_else(|| DEFAULT_SSE_ALGORITHM.to_string());
let validated = validate_ssec_params(SsecParams {
algorithm,
key: key.to_string(),
key_md5: key_md5.clone(),
})?;
let stored_md5 = metadata.get("x-amz-server-side-encryption-customer-key-md5");
verify_ssec_key_match(&validated.key_md5, stored_md5, minio_sealed_ssec_metadata(metadata))
}
/// Build an `SseError` with `InvalidRequest` (HTTP 400) for SSE-C related errors.
pub fn ssec_invalid_request(message: &str) -> SseError {
SseError {
code: S3ErrorCode::InvalidRequest,
message: message.to_string(),
source: None,
}
}
-550
View File
@@ -1,550 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Write entry points: resolve the encryption material for a new object or multipart upload.
use crate::core::audit::{envelope_master_key_version, record_managed_kms_outcome};
use crate::core::keying::context::build_object_encryption_context;
use crate::core::keying::material::EncryptionMaterial;
use crate::core::keying::{
SESSION_KEY_NEEDS_CUSTOMER_KEY, new_key_under_customer_key, new_key_under_data_key, new_session_key_under_customer_key,
};
use crate::core::request::SseKmsPrincipal;
use crate::core::s3::authorization::authorize_sse_kms_key;
use crate::core::s3::headers::reject_kms_context_location_override;
use crate::core::s3::scheme::{
DEFAULT_SSE_ALGORITHM, WriteScheme, WriteSchemeHeaders, is_managed_sse, managed_sse_type, resolve_write_scheme,
};
use crate::core::s3::ssec::{SsecParams, ssec_invalid_request, validate_ssec_params, verify_ssec_key_match};
use crate::error::SseError;
use crate::host::SseHost;
use crate::kms::provider::get_sse_dek_provider;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_kms::KmsUnavailableError;
use rustfs_policy::policy::action::KmsAction;
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::http::headers::{AMZ_ENCRYPTION_AES, AMZ_ENCRYPTION_KMS};
use rustfs_utils::http::object_encryption_keys::SSEType;
use std::collections::HashMap;
/// Request parameters for unified encryption
#[derive(Clone)]
pub struct EncryptionRequest<'a> {
/// Bucket name
pub bucket: &'a str,
/// Object key
pub key: &'a str,
/// Server-side encryption algorithm (SSE-S3 or SSE-KMS)
pub server_side_encryption: Option<String>,
/// KMS key ID (for SSE-KMS)
pub ssekms_key_id: Option<String>,
/// Optional client-provided KMS context for SSE-KMS.
pub ssekms_context: Option<HashMap<String, String>>,
/// SSE-C algorithm (customer-provided key)
pub sse_customer_algorithm: Option<String>,
/// SSE-C key (Base64-encoded)
pub sse_customer_key: Option<String>,
/// SSE-C key MD5 (Base64-encoded)
pub sse_customer_key_md5: Option<String>,
/// Content size (for metadata)
pub content_size: i64,
/// Caller the SSE-KMS key usage is authorized as. `None` marks an internal caller.
pub principal: Option<&'a SseKmsPrincipal>,
}
impl std::fmt::Debug for EncryptionRequest<'_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("EncryptionRequest")
.field("server_side_encryption", &self.server_side_encryption)
.field("content_size", &self.content_size)
.finish_non_exhaustive()
}
}
impl EncryptionRequest<'_> {
pub fn validate_complete_multipart_ssec(&self, user_defined: &HashMap<String, String>) -> Result<(), SseError> {
let request_uses_ssec =
self.sse_customer_algorithm.is_some() || self.sse_customer_key.is_some() || self.sse_customer_key_md5.is_some();
if !request_uses_ssec {
let stored_algorithm = user_defined.get("x-amz-server-side-encryption-customer-algorithm");
let stored_key_md5 = user_defined.get("x-amz-server-side-encryption-customer-key-md5");
return match (stored_algorithm, stored_key_md5) {
(None, None) => Ok(()),
(Some(algorithm), Some(key_md5))
if algorithm == DEFAULT_SSE_ALGORITHM
&& BASE64_STANDARD
.decode_to_vec(key_md5)
.is_ok_and(|decoded| decoded.len() == 16) =>
{
if rustfs_utils::http::contains_key_str(user_defined, rustfs_utils::http::SUFFIX_PLAINTEXT_CHECKSUM) {
return Err(ssec_invalid_request(
"SSE-C multipart uploads with additional checksums require the customer key for completion.",
));
}
Ok(())
}
_ => Err(ssec_invalid_request("The multipart upload contains invalid SSE-C metadata.")),
};
}
self.validate_multipart_ssec(user_defined)
}
pub fn validate_multipart_ssec(&self, user_defined: &HashMap<String, String>) -> Result<(), SseError> {
let stored_algorithm = user_defined.get("x-amz-server-side-encryption-customer-algorithm");
let stored_key_md5 = user_defined.get("x-amz-server-side-encryption-customer-key-md5");
let session_uses_ssec = stored_algorithm.is_some() || stored_key_md5.is_some();
let request_uses_ssec =
self.sse_customer_algorithm.is_some() || self.sse_customer_key.is_some() || self.sse_customer_key_md5.is_some();
if !session_uses_ssec {
return if request_uses_ssec {
Err(ssec_invalid_request(
"SSE-C parameters cannot be used for a multipart upload that was not initiated with SSE-C.",
))
} else {
Ok(())
};
}
let (Some(algorithm), Some(key), Some(key_md5)) = (
self.sse_customer_algorithm.as_ref(),
self.sse_customer_key.as_ref(),
self.sse_customer_key_md5.as_ref(),
) else {
return Err(ssec_invalid_request(
"Missing SSE-C parameters. Algorithm, customer key and customer key MD5 are all required.",
));
};
let validated = validate_ssec_params(SsecParams {
algorithm: algorithm.to_string(),
key: key.to_string(),
key_md5: key_md5.to_string(),
})?;
if stored_algorithm.map(String::as_str) != Some(validated.algorithm.as_str()) {
return Err(ssec_invalid_request(
"The provided encryption parameters did not match the multipart upload.",
));
}
// Multipart session metadata is written by our own CreateMultipartUpload,
// which always records the MD5 — the MinIO no-stored-MD5 shape cannot occur.
verify_ssec_key_match(&validated.key_md5, stored_key_md5, false)
}
}
/// **Core API**: Apply encryption based on request parameters
///
/// This function automatically routes to the appropriate encryption method:
/// - SSE-C if customer key is provided
/// - SSE-S3/SSE-KMS if server-side encryption is requested
/// - None if no encryption is requested
///
/// # Arguments
/// * `request` - Encryption request with all possible encryption parameters
///
/// # Returns
/// * `Ok(Some(material))` - Encryption should be applied with the returned material
/// * `Ok(None)` - No encryption requested
/// * `Err` - Encryption configuration error
///
/// # Example
/// ```rust,ignore
/// let request = EncryptionRequest {
/// bucket: &bucket,
/// key: &key,
/// server_side_encryption: effective_sse.as_ref(),
/// ssekms_key_id: effective_kms_key_id.as_deref(),
/// sse_customer_algorithm: sse_customer_algorithm.as_ref(),
/// sse_customer_key: sse_customer_key.as_deref(),
/// sse_customer_key_md5: sse_customer_key_md5.as_deref(),
/// content_size: actual_size,
/// part_number: None,
/// };
///
/// if let Some(material) = sse_encryption(host, request).await? {
/// reader = material.wrap_reader(reader);
/// metadata.extend(material.metadata);
/// }
/// ```
pub async fn sse_encryption(host: &dyn SseHost, request: EncryptionRequest<'_>) -> Result<Option<EncryptionMaterial>, SseError> {
let scheme = resolve_write_scheme(
host,
request.bucket,
WriteSchemeHeaders {
server_side_encryption: request.server_side_encryption,
ssekms_key_id: request.ssekms_key_id,
ssekms_context: request.ssekms_context.as_ref(),
sse_customer_algorithm: request.sse_customer_algorithm,
sse_customer_key: request.sse_customer_key,
sse_customer_key_md5: request.sse_customer_key_md5,
},
)
.await?;
let (server_side_encryption, kms_key_id) = match scheme {
None => return Ok(None),
Some(WriteScheme::SseC { algorithm, key, key_md5 }) => {
return apply_ssec_encryption_material(request.bucket, request.key, algorithm, key, key_md5, request.content_size)
.await
.map(Some);
}
Some(WriteScheme::SseS3) => (String::from(AMZ_ENCRYPTION_AES), None),
Some(WriteScheme::SseKms { key_id }) => (String::from(AMZ_ENCRYPTION_KMS), key_id),
};
apply_managed_encryption_material(
host,
ManagedEncryptionRequest {
bucket: request.bucket,
key: request.key,
server_side_encryption,
kms_key_id,
ssekms_context: request.ssekms_context,
content_size: request.content_size,
principal: request.principal,
},
)
.await
.map(Some)
}
/// **Core API**: Apply encryption based on request parameters
///
/// sse_prepare_encryption, support SSE-C, SSE-S3, SSE-KMS
pub struct PrepareEncryptionRequest<'a> {
/// Bucket name
pub bucket: &'a str,
/// Object key
pub key: &'a str,
/// Server-side encryption algorithm (SSE-S3 or SSE-KMS)
pub server_side_encryption: Option<String>,
/// KMS key ID (for SSE-KMS)
pub ssekms_key_id: Option<String>,
/// Optional client-provided KMS context for SSE-KMS.
pub ssekms_context: Option<HashMap<String, String>>,
/// SSE-C algorithm (customer-provided key)
pub sse_customer_algorithm: Option<String>,
/// SSE-C key (Base64-encoded)
pub sse_customer_key: Option<String>,
/// SSE-C key MD5 (Base64-encoded)
pub sse_customer_key_md5: Option<String>,
/// Caller the SSE-KMS key usage is authorized as. `None` marks an internal caller.
///
/// This is the only point at which a multipart upload evaluates KMS key permissions:
/// it is where the session data key is generated. `UploadPart`, `UploadPartCopy` and
/// `CompleteMultipartUpload` reuse that envelope and are not re-authorized.
pub principal: Option<&'a SseKmsPrincipal>,
}
pub async fn sse_prepare_encryption(
host: &dyn SseHost,
request: PrepareEncryptionRequest<'_>,
) -> Result<Option<EncryptionMaterial>, SseError> {
let scheme = resolve_write_scheme(
host,
request.bucket,
WriteSchemeHeaders {
server_side_encryption: request.server_side_encryption,
ssekms_key_id: request.ssekms_key_id,
ssekms_context: request.ssekms_context.as_ref(),
sse_customer_algorithm: request.sse_customer_algorithm,
sse_customer_key: request.sse_customer_key,
sse_customer_key_md5: request.sse_customer_key_md5,
},
)
.await?;
let (server_side_encryption, kms_key_id) = match scheme {
None => return Ok(None),
Some(WriteScheme::SseC { algorithm, key, key_md5 }) => {
return apply_ssec_prepare_encryption_material(request.bucket, request.key, algorithm, Some(key), key_md5)
.await
.map(Some);
}
Some(WriteScheme::SseS3) => (String::from(AMZ_ENCRYPTION_AES), None),
Some(WriteScheme::SseKms { key_id }) => (String::from(AMZ_ENCRYPTION_KMS), key_id),
};
// A session records no plaintext size: the parts decide it.
apply_managed_encryption_material(
host,
ManagedEncryptionRequest {
bucket: request.bucket,
key: request.key,
server_side_encryption,
kms_key_id,
ssekms_context: request.ssekms_context,
content_size: 0,
principal: request.principal,
},
)
.await
.map(Some)
}
async fn apply_ssec_prepare_encryption_material(
bucket: &str,
key: &str,
algorithm: String,
sse_key: Option<String>,
sse_key_md5: String,
) -> Result<EncryptionMaterial, SseError> {
// The customer key is validated here only in a mode that seals the session's
// key under it. Otherwise it is validated per part, against what the session
// records.
let customer_key = match sse_key {
Some(sse_key) if SESSION_KEY_NEEDS_CUSTOMER_KEY => Some(
validate_ssec_params(SsecParams {
algorithm: algorithm.clone(),
key: sse_key,
key_md5: sse_key_md5.clone(),
})?
.key_bytes,
),
_ => None,
};
let object_key = new_session_key_under_customer_key(customer_key, bucket, key)?;
Ok(EncryptionMaterial {
sse_type: SSEType::SseC,
server_side_encryption: String::from(AMZ_ENCRYPTION_AES),
kms_key_id: None,
algorithm,
key_bytes: object_key.key_bytes,
base_nonce: object_key.base_nonce,
encrypted_data_key: None,
customer_key_md5: Some(sse_key_md5),
original_size: None,
key_kind: object_key.key_kind,
managed_kms_context: None,
managed_sealed_key: object_key.sealed_key,
})
}
async fn apply_ssec_encryption_material(
bucket: &str,
key: &str,
algorithm: String,
sse_key: String,
sse_key_md5: String,
content_size: i64,
) -> Result<EncryptionMaterial, SseError> {
let params = SsecParams {
algorithm,
key: sse_key,
key_md5: sse_key_md5,
};
let validated = validate_ssec_params(params)?;
let object_key = new_key_under_customer_key(validated.key_bytes, bucket, key)?;
// Build metadata
Ok(EncryptionMaterial {
sse_type: SSEType::SseC,
server_side_encryption: String::from(AMZ_ENCRYPTION_AES),
kms_key_id: None,
algorithm: validated.algorithm,
key_bytes: object_key.key_bytes,
base_nonce: object_key.base_nonce,
encrypted_data_key: None,
customer_key_md5: Some(validated.key_md5),
original_size: Some(content_size),
key_kind: object_key.key_kind,
managed_kms_context: None,
managed_sealed_key: object_key.sealed_key,
})
}
/// A managed-SSE (SSE-S3 / SSE-KMS) write, once its effective scheme is known.
pub(crate) struct ManagedEncryptionRequest<'a> {
pub(crate) bucket: &'a str,
pub(crate) key: &'a str,
pub(crate) server_side_encryption: String,
pub(crate) kms_key_id: Option<String>,
pub(crate) ssekms_context: Option<HashMap<String, String>>,
pub(crate) content_size: i64,
pub(crate) principal: Option<&'a SseKmsPrincipal>,
}
pub(crate) async fn apply_managed_encryption_material(
host: &dyn SseHost,
request: ManagedEncryptionRequest<'_>,
) -> Result<EncryptionMaterial, SseError> {
// Request validation, not a KMS operation: reject before the KMS is called
// and before an audit outcome is recorded.
reject_kms_context_location_override(request.bucket, request.ssekms_context.as_ref())?;
let principal = request.principal;
let requested_sse_type = managed_sse_type(request.server_side_encryption.as_str());
let requested_key_id = request.kms_key_id.clone();
let result = apply_managed_encryption_material_inner(host, request).await;
match &result {
// The resolved key is only known on success: it may come from the request,
// the bucket default or the KMS service default. On failure the audit entry
// records what the caller asked for, which is what a reader needs to see.
Ok(material) => record_managed_kms_outcome(
principal,
material.sse_type,
material.kms_key_id.as_deref(),
|| material.encrypted_data_key.as_deref().and_then(envelope_master_key_version),
Ok(()),
),
Err(error) => record_managed_kms_outcome(principal, requested_sse_type, requested_key_id.as_deref(), || None, Err(error)),
}
result
}
pub(crate) async fn apply_managed_encryption_material_inner(
host: &dyn SseHost,
request: ManagedEncryptionRequest<'_>,
) -> Result<EncryptionMaterial, SseError> {
let ManagedEncryptionRequest {
bucket,
key,
server_side_encryption,
kms_key_id,
ssekms_context,
content_size,
principal,
} = request;
if !is_managed_sse(&server_side_encryption) {
return Err(SseError::internal(format!(
"Unsupported server-side encryption: {}",
server_side_encryption.as_str()
)));
}
let encryption_type = match server_side_encryption.as_str() {
"AES256" => SSEType::SseS3,
"aws:kms" => SSEType::SseKms,
_ => SSEType::SseS3,
};
// Determine KMS key ID to use for internal key wrapping.
let mut kms_key_candidate = kms_key_id.clone();
if kms_key_candidate.is_none() {
// Try to get default key from KMS service (if available)
if let Some(service) = host.encryption_service().await {
kms_key_candidate = service.get_default_key_id().cloned();
tracing::debug!(
default_key_id = ?kms_key_candidate,
"SSE-S3: KMS service available, default_key_id from config"
);
} else {
tracing::debug!("SSE-S3: KMS encryption service not available");
}
}
let kms_key_to_use = match (encryption_type, kms_key_candidate.clone()) {
(SSEType::SseS3, Some(kms_key_id)) => kms_key_id,
(SSEType::SseS3, None) => {
tracing::debug!("SSE-S3: no KMS key configured, falling back to \"default\" key ID");
"default".to_string()
}
(SSEType::SseKms, Some(kms_key_id)) => kms_key_id,
(SSEType::SseKms, None) => {
// Neither the request nor the bucket default named a key and no
// service default filled in. Without a service this is the same
// outage/misconfiguration the provider check below reports, so it
// must carry the same 503/400 split rather than an untyped
// internal error; with a running service that has no default key
// the caller simply has to name one.
if host.encryption_service().await.is_none() {
return Err(sse_kms_unavailable_error(kms_configured_but_unavailable(host).await));
}
return Err(SseError {
code: S3ErrorCode::InvalidRequest,
message: "SSE-KMS requires a KMS key id: the request named none and the KMS service has no default key"
.to_string(),
source: None,
});
}
_ => unreachable!("managed SSE branch only supports SSE-S3 or SSE-KMS"),
};
// The key is fully resolved here (request header, then bucket default, then the KMS
// service default), so this is the first point at which the caller can be held to the
// key it will actually be encrypted under.
authorize_sse_kms_key(host, principal, encryption_type, KmsAction::GenerateDataKeyAction, &kms_key_to_use).await?;
// A node-local master key is the explicit SSE-S3 fallback. Letting it serve
// an SSE-KMS request would persist an `aws:kms` marker and a KMS key id
// that never wrapped the data key, so refuse rather than downgrade.
//
// The refusal sits after the authorization gate so an unauthorized caller
// still sees AccessDenied whatever the KMS runtime state is, and it asks
// the resolved provider rather than a parallel availability signal,
// because the provider is what actually wraps the DEK.
let provider = match get_sse_dek_provider(host).await {
Ok(provider) => {
if matches!(encryption_type, SSEType::SseKms) && provider.wraps_dek_with_local_master_key() {
return Err(sse_kms_unavailable_error(kms_configured_but_unavailable(host).await));
}
provider
}
// With no master key set the local fallback fails with an SSE-S3-worded
// configuration error. An SSE-KMS request never asked for that provider,
// so it gets the SSE-KMS refusal instead of a message naming the wrong
// scheme.
Err(err) => {
if matches!(encryption_type, SSEType::SseKms) && host.encryption_service().await.is_none() {
return Err(sse_kms_unavailable_error(kms_configured_but_unavailable(host).await));
}
return Err(err);
}
};
let object_context = build_object_encryption_context(bucket, key, ssekms_context.as_ref());
let (data_key, encrypted_data_key) = provider.generate_sse_dek(&object_context, &kms_key_to_use).await?;
let algorithm = server_side_encryption.as_str().to_string();
let object_key = new_key_under_data_key(&data_key, encryption_type, bucket, key)?;
Ok(EncryptionMaterial {
sse_type: encryption_type,
server_side_encryption,
kms_key_id: Some(kms_key_to_use),
algorithm,
key_bytes: object_key.key_bytes,
base_nonce: object_key.base_nonce,
encrypted_data_key: Some(encrypted_data_key),
customer_key_md5: None,
original_size: Some(content_size),
key_kind: object_key.key_kind,
managed_kms_context: matches!(encryption_type, SSEType::SseKms).then_some(ssekms_context.unwrap_or_default()),
managed_sealed_key: object_key.sealed_key,
})
}
/// Whether this node has a KMS configured that is not currently serving, which
/// separates a transient outage (retryable, 503) from a deployment that never
/// configured KMS at all (a client-side configuration error, 400).
async fn kms_configured_but_unavailable(host: &dyn SseHost) -> bool {
match host.kms_service_manager() {
Some(manager) => !matches!(manager.get_status().await, rustfs_kms::KmsServiceStatus::NotConfigured),
None => false,
}
}
pub(crate) fn sse_kms_unavailable_error(configured_but_unavailable: bool) -> SseError {
if configured_but_unavailable {
return SseError::internal(KmsUnavailableError);
}
SseError {
code: S3ErrorCode::InvalidRequest,
message: "SSE-KMS requires a configured and running KMS service".to_string(),
source: None,
}
}
-176
View File
@@ -1,176 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The error every SSE operation reports, already classified for the S3 data path.
use rustfs_kms::{KmsError, KmsUnavailableError};
use rustfs_s3_types::S3ErrorCode;
type BoxError = Box<dyn std::error::Error + Send + Sync>;
/// S3 error code a KMS key that does not exist is reported under.
pub const KMS_KEY_NOT_FOUND_ERROR_CODE: &str = "KMS.NotFoundException";
pub(crate) const INTERNAL_ERROR_MESSAGE: &str = "We encountered an internal error, please try again.";
const SERVICE_UNAVAILABLE_MESSAGE: &str = "The service is unavailable. Please retry.";
const ACCESS_DENIED_MESSAGE: &str = "Access Denied.";
/// A failed SSE operation, carrying the S3 error code and message the caller answers with.
///
/// `source` keeps the underlying failure for logs and for classification by
/// callers; it is never part of the response.
#[derive(Debug)]
pub struct SseError {
pub code: S3ErrorCode,
pub message: String,
pub source: Option<BoxError>,
}
impl SseError {
/// Classify a failure that did not originate from the request itself.
///
/// A [`KmsUnavailableError`] is a retryable `ServiceUnavailable`, a
/// [`KmsError`] follows [`data_plane_kms_error`], and everything else is an
/// `InternalError` whose detail stays in `source` and out of the response.
pub fn internal(error: impl Into<BoxError>) -> Self {
let source = error.into();
let (code, message) = if source.downcast_ref::<KmsUnavailableError>().is_some() {
(S3ErrorCode::ServiceUnavailable, SERVICE_UNAVAILABLE_MESSAGE.to_string())
} else if let Some(classified) = source.downcast_ref::<KmsError>().and_then(data_plane_kms_error) {
classified
} else {
(S3ErrorCode::InternalError, INTERNAL_ERROR_MESSAGE.to_string())
};
Self {
code,
message,
source: Some(source),
}
}
}
impl std::fmt::Display for SseError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.message)
}
}
impl std::error::Error for SseError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
self.source.as_deref().map(|source| source as _)
}
}
/// Map a KMS failure that surfaced on the S3 data path to its S3 error code.
///
/// The contract deliberately differs from the admin key-lifecycle handlers:
/// there a key id is the resource being addressed, so a missing key is `404`.
/// Here the key id arrives inside a request header or a bucket default, so a
/// key that is missing, disabled, or otherwise unusable is configuration the
/// caller has to correct — AWS answers `400`, and reporting `500` instead both
/// misfiles the failure as a server fault and makes SDKs back off and retry a
/// request that cannot succeed.
///
/// `None` keeps the caller's fallthrough, which is the `500` that integrity
/// faults — damaged, unreadable, or unknown-format key material — must keep.
///
/// Messages either name what the caller asked for or stay generic; detail that
/// belongs to the deployment rather than the request stays in `source`, which
/// the caller attaches and the logs retain.
pub fn data_plane_kms_error(error: &KmsError) -> Option<(S3ErrorCode, String)> {
use KmsError as Kms;
match error {
Kms::KeyNotFound { key_id } => Some((
S3ErrorCode::Custom(KMS_KEY_NOT_FOUND_ERROR_CODE.into()),
format!("KMS key not found: {key_id}"),
)),
Kms::KeyVersionNotFound { key_id, version } => Some((
S3ErrorCode::Custom(KMS_KEY_NOT_FOUND_ERROR_CODE.into()),
format!("KMS key version {version} not found for key {key_id}"),
)),
// The key exists but its state forbids the operation (disabled,
// pending deletion). AWS treats an invalid key state as a request
// error, not a server fault.
Kms::InvalidOperation { .. } => Some((S3ErrorCode::InvalidRequest, error.to_string())),
// A policy decision needs a human, so it must be distinguishable from
// the transient classes an SDK retries.
Kms::AccessDenied { .. } => Some((S3ErrorCode::AccessDenied, ACCESS_DENIED_MESSAGE.to_string())),
// Request-side faults: what was asked for cannot be served as asked.
Kms::ContextMismatch { .. }
| Kms::InvalidKey { .. }
| Kms::ValidationError { .. }
| Kms::UnsupportedAlgorithm { .. }
| Kms::InvalidKeySize { .. } => Some((S3ErrorCode::InvalidRequest, error.to_string())),
// A deployment whose KMS configuration cannot serve the request (for
// example no key named and no default). Actionable, but the detail
// describes the deployment, so it stays out of the response body.
Kms::ConfigurationError { .. } => {
Some((S3ErrorCode::InvalidRequest, "The KMS configuration cannot serve this request".to_string()))
}
// Transient: worth retrying, and must be counted against availability
// rather than against the caller. `IoError` belongs here because it is
// how a backend reports that its key store itself was unreachable —
// rustfs/rustfs#7470 separated that from a missing key precisely so the
// two stop looking alike, and leaving it on the 500 fallthrough would
// erase that distinction again at the S3 boundary.
Kms::BackendError { .. }
| Kms::IoError { .. }
| Kms::OperationTimedOut { .. }
| Kms::OperationCancelled { .. }
| Kms::CredentialsUnavailable { .. }
| Kms::CacheError { .. } => Some((S3ErrorCode::ServiceUnavailable, SERVICE_UNAVAILABLE_MESSAGE.to_string())),
// A permanent gap in the configured backend, never a missing resource.
Kms::UnsupportedCapability { .. } => Some((S3ErrorCode::NotImplemented, error.to_string())),
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn internal_keeps_request_free_failures_generic() {
let error = SseError::internal("Failed to decode IV: bad base64");
assert_eq!(error.code, S3ErrorCode::InternalError);
assert_eq!(error.message, INTERNAL_ERROR_MESSAGE);
assert!(
error
.source
.as_ref()
.is_some_and(|source| source.to_string().contains("Failed to decode IV"))
);
}
#[test]
fn internal_reports_an_unavailable_kms_as_retryable() {
let error = SseError::internal(KmsUnavailableError);
assert_eq!(error.code, S3ErrorCode::ServiceUnavailable);
assert_eq!(error.message, SERVICE_UNAVAILABLE_MESSAGE);
}
#[test]
fn internal_classifies_kms_errors_for_the_data_path() {
let missing = SseError::internal(KmsError::key_not_found("absent-key"));
assert_eq!(missing.code, S3ErrorCode::Custom(KMS_KEY_NOT_FOUND_ERROR_CODE.into()));
assert_eq!(missing.message, "KMS key not found: absent-key");
let damaged = SseError::internal(KmsError::cryptographic_error("decrypt", "authentication failed"));
assert_eq!(damaged.code, S3ErrorCode::InternalError);
assert_eq!(damaged.message, INTERNAL_ERROR_MESSAGE);
}
}
-85
View File
@@ -1,85 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! What SSE needs from the server it runs in.
//!
//! This crate reaches the running server only through [`SseHost`]: it holds no
//! handle to the storage engine, to IAM, or to the bucket metadata system. Every
//! entry point that consults the server takes the host as its first argument, so
//! a caller cannot reach key selection without saying which server it speaks for.
use crate::{SseError, SseKmsPrincipal};
use async_trait::async_trait;
use rustfs_kms::KmsServiceManager;
use rustfs_kms::service::ObjectEncryptionService;
use rustfs_policy::policy::action::KmsAction;
use std::sync::Arc;
/// The server capabilities SSE key resolution depends on.
#[async_trait]
pub trait SseHost: Send + Sync {
/// The KMS object-encryption service currently serving, if one is.
///
/// Resolved on every call: the KMS backend can be reconfigured while the
/// server runs, so a service handle must not outlive the operation it serves.
async fn encryption_service(&self) -> Option<Arc<ObjectEncryptionService>>;
/// The KMS service manager, when the server has one.
///
/// Only its status is read, to tell a configured KMS that is not serving
/// (retryable) from a deployment that never configured one (a request error).
fn kms_service_manager(&self) -> Option<Arc<KmsServiceManager>>;
/// The bucket's default-encryption configuration.
///
/// `Ok(None)` means the bucket has no encryption configuration.
///
/// # Errors
///
/// Any failure to read the configuration, already classified for the S3
/// response. A write must not proceed as if the bucket had no default when
/// the configuration could not be read.
async fn bucket_encryption(&self, bucket: &str) -> Result<Option<BucketEncryption>, SseError>;
/// The authorizer deciding per-key KMS permissions on the data path.
fn kms_key_authorizer(&self) -> &dyn KmsKeyAuthorizer;
}
/// A bucket's stored encryption configuration, reduced to what a write consults.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct BucketEncryption {
/// The default-encryption rule of the first configuration rule, if it has one.
pub default_rule: Option<BucketDefaultEncryption>,
}
/// The `ApplyServerSideEncryptionByDefault` element of a bucket encryption rule.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct BucketDefaultEncryption {
/// The stored algorithm, exactly as persisted (`AES256`, `aws:kms`, or an unknown value).
pub sse_algorithm: String,
pub kms_master_key_id: Option<String>,
}
/// Decides whether a principal may use a specific KMS key on the data path.
///
/// Behind a trait so tests can pin the allow/deny contract of the SSE call sites
/// without a running IAM subsystem.
#[async_trait]
pub trait KmsKeyAuthorizer: Send + Sync {
/// # Errors
///
/// When the decision itself could not be made, for example because the
/// policy subsystem is not ready. A denial is `Ok(false)`.
async fn is_allowed(&self, principal: &SseKmsPrincipal, action: KmsAction, key_id: &str) -> Result<bool, SseError>;
}
-180
View File
@@ -1,180 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The provider that wraps and unwraps data keys through the KMS service.
use crate::core::audit::KmsDataPlaneFailure;
use crate::error::SseError;
use crate::host::SseHost;
use crate::kms::provider::SseDekProvider;
use async_trait::async_trait;
use rustfs_kms::types::ObjectEncryptionContext;
use rustfs_kms::{DataKey, KmsUnavailableError};
use rustfs_s3_types::S3ErrorCode;
use std::sync::Arc;
/// Production KMS-backed DEK provider
/// Resolves the latest ObjectEncryptionService on each call.
pub(crate) struct KmsSseDekProvider<'a> {
service_source: KmsServiceSource<'a>,
}
/// Where a [`KmsSseDekProvider`] finds the service that is serving right now.
enum KmsServiceSource<'a> {
Host(&'a dyn SseHost),
#[cfg(test)]
Manager(Arc<rustfs_kms::KmsServiceManager>),
}
pub(crate) fn kms_operation_error(error: rustfs_kms::KmsError) -> SseError {
let class = rustfs_kms::audit::error_class(&error);
let mut api_error = SseError::internal(error);
// Wrap rather than replace the source so the original chain stays intact for
// logging; the wrapper only adds the class the audit attachment needs.
let source = api_error.source.take();
api_error.source = Some(Box::new(KmsDataPlaneFailure { class, source }));
api_error
}
/// Classification for a failed data-key unwrap.
///
/// An AEAD failure stays `500` (the envelope is an integrity fault, not a
/// request a retry or a different header can fix), but the generic internal
/// error text hides the one diagnosis an operator needs: the configured
/// backend holds different key material under this key id than the one that
/// wrapped the object, typically after re-creating a key of the same name or
/// switching backends.
pub(crate) fn kms_unwrap_error(error: rustfs_kms::KmsError) -> SseError {
let unwrap_rejected = matches!(error, rustfs_kms::KmsError::CryptographicError { .. });
let mut api_error = kms_operation_error(error);
if unwrap_rejected && api_error.code == S3ErrorCode::InternalError {
api_error.message = "The object's data key envelope could not be unwrapped by the configured KMS backend: the \
key material under this key id differs from the one that wrapped it, or the envelope is damaged"
.to_string();
}
api_error
}
impl<'a> KmsSseDekProvider<'a> {
/// Create a new KMS-backed provider
pub async fn new(host: &'a dyn SseHost) -> Result<Self, SseError> {
let provider = Self {
service_source: KmsServiceSource::Host(host),
};
provider
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
Ok(provider)
}
#[cfg(test)]
pub(crate) async fn new_with_service_manager(service_manager: Arc<rustfs_kms::KmsServiceManager>) -> Result<Self, SseError> {
let provider = Self {
service_source: KmsServiceSource::Manager(service_manager),
};
provider
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
Ok(provider)
}
async fn current_service(&self) -> Option<Arc<rustfs_kms::service::ObjectEncryptionService>> {
match &self.service_source {
KmsServiceSource::Host(host) => host.encryption_service().await,
#[cfg(test)]
KmsServiceSource::Manager(service_manager) => service_manager.get_encryption_service().await,
}
}
}
#[async_trait]
impl SseDekProvider for KmsSseDekProvider<'_> {
async fn generate_sse_dek(
&self,
context: &ObjectEncryptionContext,
kms_key_id: &str,
) -> Result<(DataKey, Vec<u8>), SseError> {
let kms_key_option = Some(kms_key_id.to_string());
let service = self
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
let (data_key, encrypted_data_key) = service
.create_data_key(&kms_key_option, context)
.await
.map_err(kms_operation_error)?;
Ok((data_key, encrypted_data_key))
}
async fn reads_bare_data_key_ciphertext(&self) -> bool {
self.current_service()
.await
.is_some_and(|service| service.reads_bare_data_key_ciphertext())
}
async fn rewrap_sse_dek(
&self,
encrypted_dek: &[u8],
context: &ObjectEncryptionContext,
) -> Result<rustfs_kms::types::RewrapDataKeyResponse, SseError> {
let service = self
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
service
.rewrap_data_key(encrypted_dek, context)
.await
.map_err(kms_operation_error)
}
async fn decrypt_sse_dek(
&self,
encrypted_dek: &[u8],
_kms_key_id: &str,
context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
let service = self
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
let data_key = service
.decrypt_data_key(encrypted_dek, context)
.await
.map_err(kms_unwrap_error)?;
Ok(data_key.plaintext_key)
}
#[cfg(feature = "rio-v2")]
async fn decrypt_legacy_sse_dek(
&self,
encrypted_dek: &[u8],
_kms_key_id: &str,
_context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
let service = self
.current_service()
.await
.ok_or_else(|| SseError::internal(KmsUnavailableError))?;
let data_key = service
.decrypt_legacy_data_key(encrypted_dek)
.await
.map_err(kms_unwrap_error)?;
Ok(data_key.plaintext_key)
}
}
-323
View File
@@ -1,323 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The provider that wraps data keys with a node-local master key when no KMS is configured.
use crate::error::SseError;
#[cfg(feature = "rio-v2")]
use crate::kms::minio::{decrypt_minio_kms_data_key, minio_kms_associated_data};
use crate::kms::provider::SseDekProvider;
use aes_gcm::aead::{Aead, KeyInit};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use async_trait::async_trait;
use base64_simd::STANDARD as BASE64_STANDARD;
use rand::Rng;
use rustfs_kms::DataKey;
use rustfs_kms::types::ObjectEncryptionContext;
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::get_env_opt_str;
use serde::{Deserialize, Serialize};
/// Local SSE DEK provider for deployments without a KMS.
///
/// Uses `RUSTFS_SSE_S3_MASTER_KEY` (base64-encoded 32-byte key) to wrap
/// data-encryption keys with AES-256-GCM. This is the production fallback
/// when no KMS service is configured.
///
/// # Environment Variable
///
/// ```text
/// RUSTFS_SSE_S3_MASTER_KEY=<base64_encoded_32_byte_key>
/// ```
pub struct LocalSseDekProvider {
master_key: [u8; 32],
}
pub(crate) const LOCAL_SSE_DEK_FORMAT_VERSION: u8 = 1;
/// Environment variable that carries the local SSE-S3 master key.
pub(crate) const ENV_SSE_S3_MASTER_KEY: &str = "RUSTFS_SSE_S3_MASTER_KEY";
/// The configured local master key, still encoded; `None` when unset or blank.
fn local_master_key_env_value() -> Option<String> {
get_env_opt_str(ENV_SSE_S3_MASTER_KEY).filter(|value| !value.trim().is_empty())
}
/// Whether the operator configured a local SSE-S3 master key at all.
///
/// Says nothing about whether the value is usable; loading it decides that.
pub(crate) fn local_master_key_configured() -> bool {
local_master_key_env_value().is_some()
}
#[cfg(feature = "rio-v2")]
/// Returns true when a managed-SSE data key is one RustFS itself wrote.
///
/// Both RustFS envelope shapes are strict JSON — the KMS envelope
/// ([`rustfs_kms::is_data_key_envelope`]) and the local provider's
/// [`LocalSseDekEnvelope`], whose `deny_unknown_fields` keeps it from accepting
/// anything else. Recognition is deliberately positive: an unrecognized payload
/// is left to MinIO's decoder rather than guessed at, and neither decoder is
/// ever handed the other's format.
pub(crate) fn is_rustfs_managed_data_key(encrypted_dek: &[u8]) -> bool {
if rustfs_kms::is_data_key_envelope(encrypted_dek) {
return true;
}
std::str::from_utf8(encrypted_dek)
.ok()
.is_some_and(|text| serde_json::from_str::<LocalSseDekEnvelope<'_>>(text).is_ok())
}
#[derive(Debug, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub(crate) struct LocalSseDekEnvelope<'a> {
pub(crate) version: u8,
pub(crate) nonce: &'a str,
ciphertext: &'a str,
}
/// Whether [`LocalSseDekProvider`] could have written these bytes.
///
/// Covers both shapes [`LocalSseDekProvider::decrypt_dek`] accepts — anything
/// presented as a JSON object, so a damaged local envelope keeps its own
/// diagnosis, and the deprecated `nonce:ciphertext` base64 pair. Everything
/// else is bytes the local provider can only reject.
pub(crate) fn is_local_sse_dek(encrypted_dek: &[u8]) -> bool {
let Ok(text) = std::str::from_utf8(encrypted_dek) else {
return false;
};
if text.trim_start().starts_with('{') {
return true;
}
let is_base64 = |part: &str| BASE64_STANDARD.decode_to_vec(part).is_ok();
text.split_once(':')
.is_some_and(|(nonce, ciphertext)| is_base64(nonce) && is_base64(ciphertext))
}
/// Test-only alias so existing test code that references `TestSseDekProvider`
/// continues to compile without changes.
#[cfg(any(test, feature = "test-util"))]
#[allow(non_camel_case_types)]
pub type TestSseDekProvider = LocalSseDekProvider;
/// Parse the base64-encoded 32-byte master key from `__RUSTFS_SSE_SIMPLE_CMK`.
///
/// Returns an error (never crashes) for a missing, non-base64, wrong-length, or
/// all-zero key so callers on the request path can fail the request instead of
/// taking the whole server down (backlog#806).
#[cfg(any(test, feature = "test-util"))]
pub(crate) fn parse_simple_sse_cmk(cmk_value: &str) -> Result<[u8; 32], SseError> {
let trimmed = cmk_value.trim();
if trimmed.is_empty() {
return Err(SseError::internal(
"SSE simple mode requires __RUSTFS_SSE_SIMPLE_CMK to be set to a base64-encoded 32-byte key",
));
}
let decoded = BASE64_STANDARD
.decode_to_vec(trimmed)
.map_err(|e| SseError::internal(format!("__RUSTFS_SSE_SIMPLE_CMK must be valid base64: {e}")))?;
let master_key: [u8; 32] = decoded.try_into().map_err(|v: Vec<u8>| {
SseError::internal(format!("__RUSTFS_SSE_SIMPLE_CMK must decode to exactly 32 bytes, got {} bytes", v.len()))
})?;
if master_key == [0u8; 32] {
return Err(SseError::internal("__RUSTFS_SSE_SIMPLE_CMK must not be an all-zero key"));
}
Ok(master_key)
}
impl LocalSseDekProvider {
/// Create a LocalSseDekProvider with a predefined key (for testing)
#[cfg(any(test, feature = "test-util"))]
pub fn new_with_key(master_key: [u8; 32]) -> Self {
Self { master_key }
}
/// Create a TestSseDekProvider from `__RUSTFS_SSE_SIMPLE_CMK` (test-only).
#[cfg(any(test, feature = "test-util"))]
pub fn new() -> Result<Self, SseError> {
let cmk_value = std::env::var("__RUSTFS_SSE_SIMPLE_CMK").unwrap_or_default();
// A missing/invalid key must surface as a request error, never crash the
// whole server: `LocalSseDekProvider::new` is reached from the SSE request
// path (get_sse_dek_provider), so `process::exit(1)` here turned a bad
// `__RUSTFS_SSE_SIMPLE_CMK` into a process crash-loop DoS (backlog#806).
let master_key = parse_simple_sse_cmk(&cmk_value)?;
tracing::info!("Successfully loaded SSE master key (32 bytes) from __RUSTFS_SSE_SIMPLE_CMK");
Ok(Self { master_key })
}
/// Create a local SSE DEK provider for SSE-S3 when KMS is not configured.
/// Requires RUSTFS_SSE_S3_MASTER_KEY to be a valid base64-encoded 32-byte key.
///
/// The failures here are server configuration problems, not internal
/// faults: surface them as `InvalidRequest` (HTTP 400) so a managed-SSE
/// request against an unconfigured server does not report 500 (rustfs#4844).
pub fn new_from_env() -> Result<Self, SseError> {
fn sse_not_configured(message: impl Into<String>) -> SseError {
SseError {
code: S3ErrorCode::InvalidRequest,
message: message.into(),
source: None,
}
}
let Some(raw_value) = local_master_key_env_value() else {
return Err(sse_not_configured(
"SSE-S3 requires RUSTFS_SSE_S3_MASTER_KEY to be set to a base64-encoded 32-byte key when KMS is not configured",
));
};
let decoded = BASE64_STANDARD.decode_to_vec(raw_value.trim()).map_err(|err| {
sse_not_configured(format!(
"RUSTFS_SSE_S3_MASTER_KEY must be valid base64 for SSE-S3 when KMS is not configured: {err}"
))
})?;
let master_key: [u8; 32] = decoded.try_into().map_err(|_| {
sse_not_configured("RUSTFS_SSE_S3_MASTER_KEY must decode to exactly 32 bytes for SSE-S3 when KMS is not configured")
})?;
Ok(Self { master_key })
}
// Simple encryption of DEK
pub fn encrypt_dek(dek: [u8; 32], cmk_value: [u8; 32]) -> Result<String, SseError> {
// Use AES-256-GCM to encrypt DEK
let key = Key::<Aes256Gcm>::from(cmk_value);
let cipher = Aes256Gcm::new(&key);
let mut nonce_bytes = [0u8; 12];
rand::rng().fill_bytes(&mut nonce_bytes);
let nonce = Nonce::from(nonce_bytes);
let ciphertext = cipher
.encrypt(&nonce, dek.as_slice())
.map_err(|_| SseError::internal("Failed to encrypt DEK"))?;
let nonce = BASE64_STANDARD.encode_to_string(nonce);
let ciphertext = BASE64_STANDARD.encode_to_string(ciphertext);
serde_json::to_string(&LocalSseDekEnvelope {
version: LOCAL_SSE_DEK_FORMAT_VERSION,
nonce: &nonce,
ciphertext: &ciphertext,
})
.map_err(|e| SseError::internal(format!("Failed to serialize encrypted DEK: {e}")))
}
// Simple decryption of DEK
pub fn decrypt_dek(encrypted_dek: &str, cmk_value: [u8; 32]) -> Result<[u8; 32], SseError> {
let envelope = serde_json::from_str::<LocalSseDekEnvelope<'_>>(encrypted_dek);
let (nonce, ciphertext) = match envelope {
Ok(envelope) => {
if envelope.version != LOCAL_SSE_DEK_FORMAT_VERSION {
return Err(SseError::internal(format!(
"Unsupported encrypted DEK format version: {}",
envelope.version
)));
}
(envelope.nonce, envelope.ciphertext)
}
Err(json_error) if encrypted_dek.trim_start().starts_with('{') => {
return Err(SseError::internal(format!("Invalid encrypted DEK JSON format: {json_error}")));
}
Err(_) => {
// DEPRECATED: read-only compatibility for persisted colon-delimited DEKs.
// RUSTFS_COMPAT_TODO(sse-local-dek-json-v1): Remove after all supported upgrades have rewritten legacy DEKs.
let Some((nonce, ciphertext)) = encrypted_dek.split_once(':') else {
return Err(SseError::internal("Invalid encrypted DEK format"));
};
if ciphertext.contains(':') {
return Err(SseError::internal("Invalid encrypted DEK format"));
}
(nonce, ciphertext)
}
};
let nonce_vec = BASE64_STANDARD
.decode_to_vec(nonce)
.map_err(|_| SseError::internal("Invalid nonce format"))?;
let ciphertext = BASE64_STANDARD
.decode_to_vec(ciphertext)
.map_err(|_| SseError::internal("Invalid ciphertext format"))?;
let key = Key::<Aes256Gcm>::from(cmk_value);
let cipher = Aes256Gcm::new(&key);
let nonce_array: [u8; 12] = nonce_vec.try_into().map_err(|_| SseError::internal("Invalid nonce length"))?;
let nonce = Nonce::from(nonce_array);
let plaintext = cipher
.decrypt(&nonce, ciphertext.as_slice())
.map_err(|e| SseError::internal(format!("Failed to decrypt DEK: {e}")))?;
let dek: [u8; 32] = plaintext
.try_into()
.map_err(|_| SseError::internal("Decrypted DEK has invalid length"))?;
Ok(dek)
}
}
#[async_trait]
impl SseDekProvider for LocalSseDekProvider {
fn wraps_dek_with_local_master_key(&self) -> bool {
true
}
async fn generate_sse_dek(
&self,
_context: &ObjectEncryptionContext,
_kms_key_id: &str,
) -> Result<(DataKey, Vec<u8>), SseError> {
// Generate a 32-byte array as data key
let mut dek = [0u8; 32];
rand::rng().fill_bytes(&mut dek);
// Generate a 12-byte array as IV
let mut nonce = [0u8; 12];
rand::rng().fill_bytes(&mut nonce);
// Encrypt data key with master key
let encrypted_dek = Self::encrypt_dek(dek, self.master_key)?;
// Return data key and IV
Ok((
DataKey {
plaintext_key: dek,
nonce,
},
encrypted_dek.into_bytes(),
))
}
async fn decrypt_sse_dek(
&self,
encrypted_dek: &[u8],
_kms_key_id: &str,
_context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
// Decrypt data key with master key
let encrypted_dek_str =
std::str::from_utf8(encrypted_dek).map_err(|_| SseError::internal("Invalid UTF-8 in encrypted DEK"))?;
let dek = Self::decrypt_dek(encrypted_dek_str, self.master_key)?;
Ok(dek)
}
#[cfg(feature = "rio-v2")]
async fn decrypt_minio_sse_dek(
&self,
encrypted_dek: &[u8],
_kms_key_id: &str,
context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
let aad = minio_kms_associated_data(context)?;
decrypt_minio_kms_data_key(encrypted_dek, &self.master_key, &aad)
}
}
-151
View File
@@ -1,151 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Reading data keys sealed by MinIO's builtin KMS.
use crate::core::keying::context::canonical_kms_bucket_path;
use crate::core::keying::object::HmacSha256;
use crate::error::SseError;
use base64_simd::STANDARD as BASE64_STANDARD;
use hmac::Mac;
use rustfs_kms::types::ObjectEncryptionContext;
use serde::Deserialize;
/// Associated data MinIO binds when sealing a data key.
///
/// MinIO passes the object's encryption context as the AEAD's associated data,
/// serialized as canonical JSON with sorted keys — the same canonicalization
/// [`rustfs_kms::context_aad`] performs, which is why the context RustFS
/// already rebuilds for the read can be reused verbatim. For SSE-S3 that
/// context is `{bucket: "bucket/object"}`; for SSE-KMS it is whatever the
/// request supplied, recovered from the stored MinIO context header.
pub(crate) fn minio_kms_associated_data(context: &ObjectEncryptionContext) -> Result<Vec<u8>, SseError> {
let mut ctx = context.encryption_context.clone();
ctx.entry(context.bucket.clone())
.or_insert_with(|| canonical_kms_bucket_path(&context.bucket, &context.object_key));
rustfs_kms::context_aad(&ctx).map_err(|e| SseError::internal(format!("Failed to canonicalize MinIO KMS context: {e}")))
}
/// MinIO's builtin-KMS ciphertext in its JSON encoding.
///
/// Deliberately its own type rather than a relaxation of
/// [`LocalSseDekEnvelope`]: widening that envelope's `deny_unknown_fields`
/// to admit this shape would also admit malformed RustFS envelopes, which
/// backlog#1567 requires to keep failing closed.
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
struct MinioKmsCiphertextJson {
aead: String,
#[allow(
dead_code,
reason = "present in MinIO's encoding; the key is identified by metadata instead"
)]
#[serde(default)]
id: String,
iv: String,
nonce: String,
bytes: String,
}
/// Bytes of trailing randomness every MinIO builtin-KMS ciphertext carries:
/// a 16-byte IV followed by a 12-byte nonce, *after* the sealed bytes.
const MINIO_KMS_RANDOM_LEN: usize = 28;
const MINIO_KMS_IV_LEN: usize = 16;
const MINIO_KMS_AEAD_AES_GCM: &str = "AES-256-GCM-HMAC-SHA-256";
const MINIO_KMS_AEAD_CHACHA20: &str = "ChaCha20Poly1305";
/// Unwrap a data key sealed by MinIO's builtin (static-secret) KMS.
///
/// The wire format is `sealed_bytes || iv[16] || nonce[12]` — the randomness
/// trails the ciphertext rather than leading it, and MinIO's own decoder
/// normalizes its legacy JSON encoding into exactly that byte order before
/// opening it (`internal/kms/secret-key.go`, `parseCiphertext`). A raw
/// (non-JSON) ciphertext is AES-256-GCM by definition there; the JSON form
/// names its algorithm.
///
/// The sealing key is derived per ciphertext rather than being the master key:
/// `HMAC-SHA256(master, iv)` for AES-256-GCM, `HChaCha20(master, iv)` for
/// ChaCha20-Poly1305. The encryption context is bound as associated data.
pub(crate) fn decrypt_minio_kms_data_key(encrypted_dek: &[u8], master_key: &[u8; 32], aad: &[u8]) -> Result<[u8; 32], SseError> {
let (body, algorithm) = match std::str::from_utf8(encrypted_dek) {
// MinIO only treats a payload as JSON when it both starts and ends like
// an object, and falls back to the raw layout when it does not parse —
// mirrored here so a ciphertext that merely looks like JSON is not
// rejected outright.
Ok(text)
if text.starts_with('{')
&& text.ends_with('}')
&& let Ok(json) = serde_json::from_str::<MinioKmsCiphertextJson>(text) =>
{
let decode = |what: &str, value: &str| -> Result<Vec<u8>, SseError> {
BASE64_STANDARD
.decode_to_vec(value)
.map_err(|e| SseError::internal(format!("Invalid MinIO KMS {what}: {e}")))
};
let mut body = decode("ciphertext", &json.bytes)?;
body.extend_from_slice(&decode("iv", &json.iv)?);
body.extend_from_slice(&decode("nonce", &json.nonce)?);
(body, json.aead)
}
_ => (encrypted_dek.to_vec(), MINIO_KMS_AEAD_AES_GCM.to_string()),
};
if body.len() <= MINIO_KMS_RANDOM_LEN {
return Err(SseError::internal("MinIO KMS ciphertext is too short to carry its IV and nonce"));
}
let (sealed, random) = body.split_at(body.len() - MINIO_KMS_RANDOM_LEN);
let (iv, nonce) = random.split_at(MINIO_KMS_IV_LEN);
let plaintext = match algorithm.as_str() {
MINIO_KMS_AEAD_AES_GCM => {
use aes_gcm::{Aes256Gcm, KeyInit, aead::Aead};
let mut mac = HmacSha256::new_from_slice(master_key)
.map_err(|_| SseError::internal("MinIO KMS sealing key derivation failed"))?;
mac.update(iv);
let sealing_key: [u8; 32] = mac.finalize().into_bytes().into();
let cipher = Aes256Gcm::new_from_slice(&sealing_key)
.map_err(|_| SseError::internal("MinIO KMS sealing key is not a valid AES-256 key"))?;
let nonce = aes_gcm::Nonce::try_from(nonce).map_err(|_| SseError::internal("MinIO KMS nonce is not 12 bytes"))?;
cipher.decrypt(&nonce, aes_gcm::aead::Payload { msg: sealed, aad })
}
MINIO_KMS_AEAD_CHACHA20 => {
use chacha20poly1305::{KeyInit, XChaCha20Poly1305, aead::Aead};
// MinIO derives this branch's key with HChaCha20 over the 16-byte
// IV, which is exactly XChaCha20-Poly1305's own construction, so the
// extended-nonce cipher does the derivation rather than hand-rolling it.
let mut extended = Vec::with_capacity(MINIO_KMS_IV_LEN + nonce.len());
extended.extend_from_slice(iv);
extended.extend_from_slice(nonce);
let cipher = XChaCha20Poly1305::new_from_slice(master_key)
.map_err(|_| SseError::internal("MinIO KMS master key is not a valid ChaCha20 key"))?;
let nonce = chacha20poly1305::XNonce::try_from(extended.as_slice())
.map_err(|_| SseError::internal("MinIO KMS extended nonce is not 24 bytes"))?;
cipher.decrypt(&nonce, chacha20poly1305::aead::Payload { msg: sealed, aad })
}
other => {
return Err(SseError::internal(format!("Unsupported MinIO KMS AEAD algorithm: {other}")));
}
}
// An AEAD failure here is authentication, not a decode slip: a wrong master
// key, a tampered ciphertext, and an encryption context that does not match
// what sealed it all land here and must all fail closed.
.map_err(|_| SseError::internal("MinIO KMS data key failed authentication"))?;
plaintext
.try_into()
.map_err(|value: Vec<u8>| SseError::internal(format!("MinIO KMS data key must be 32 bytes, got {}", value.len())))
}
-21
View File
@@ -1,21 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Data-key providers: where a data key is wrapped and unwrapped.
pub(crate) mod kms_backed;
pub(crate) mod local;
#[cfg(feature = "rio-v2")]
pub(crate) mod minio;
pub(crate) mod provider;
-288
View File
@@ -1,288 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The data-key provider interface and the per-request choice between the KMS and the local master key.
use crate::error::SseError;
use crate::host::SseHost;
use crate::kms::kms_backed::KmsSseDekProvider;
use crate::kms::local::LocalSseDekProvider;
use async_trait::async_trait;
use rustfs_kms::DataKey;
use rustfs_kms::types::ObjectEncryptionContext;
use std::sync::{Arc, LazyLock, RwLock};
use tracing::debug;
/// Trait for SSE data encryption key management
/// Abstracts the source of encryption keys (KMS, test provider, etc.)
#[async_trait]
pub trait SseDekProvider: Send + Sync {
/// Generate an SSE data encryption key
async fn generate_sse_dek(&self, context: &ObjectEncryptionContext, kms_key_id: &str)
-> Result<(DataKey, Vec<u8>), SseError>;
/// Decrypt an SSE data encryption key (returns only plaintext key, nonce should be read from metadata)
async fn decrypt_sse_dek(
&self,
encrypted_dek: &[u8],
kms_key_id: &str,
context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError>;
/// Whether this provider may own a stored data key that is not a RustFS
/// envelope — see `KmsBackend::reads_bare_data_key_ciphertext`.
async fn reads_bare_data_key_ciphertext(&self) -> bool {
false
}
/// Re-wrap a KMS-wrapped DEK envelope onto its master key's current
/// version without exposing the plaintext DEK to the caller.
///
/// Defaults to refusing: only the KMS-backed provider can rewrap, and a
/// provider that cannot must say so rather than hand back the input as if
/// it had been re-protected.
async fn rewrap_sse_dek(
&self,
_encrypted_dek: &[u8],
_context: &ObjectEncryptionContext,
) -> Result<rustfs_kms::types::RewrapDataKeyResponse, SseError> {
Err(SseError::internal("This DEK provider cannot rewrap KMS-wrapped data keys"))
}
/// Whether this provider wraps data keys with a node-local master key
/// instead of a KMS service.
///
/// SSE-KMS must never be served by such a provider: the stored object would
/// claim `aws:kms` and name a KMS key id that never wrapped anything.
/// Defaults to false so only the local fallback has to declare itself.
fn wraps_dek_with_local_master_key(&self) -> bool {
false
}
/// Decrypt a DEK from positively identified legacy managed metadata.
#[cfg(feature = "rio-v2")]
async fn decrypt_legacy_sse_dek(
&self,
encrypted_dek: &[u8],
kms_key_id: &str,
context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
self.decrypt_sse_dek(encrypted_dek, kms_key_id, context).await
}
/// Unwrap a data key that MinIO's builtin KMS sealed.
///
/// A separate entry point rather than a shape sniff inside
/// [`Self::decrypt_sse_dek`]: the caller already knows the object carries a
/// MinIO sealed key, and MinIO's raw ciphertext is unstructured bytes that
/// no parser can reliably tell apart from anything else. Routing on the
/// caller's knowledge keeps a RustFS envelope from ever reaching MinIO's
/// decoder, and vice versa.
///
/// Defaults to refusing: only a provider holding the MinIO master secret
/// can serve these, and a provider that cannot must fail rather than fall
/// back to a decoder that would misread the bytes.
#[cfg(feature = "rio-v2")]
async fn decrypt_minio_sse_dek(
&self,
_encrypted_dek: &[u8],
_kms_key_id: &str,
_context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
Err(SseError::internal(
"This KMS provider cannot unwrap a data key sealed by MinIO's builtin KMS",
))
}
}
/// Unwrap the data key of an object that carries a sealed object key but no
/// RustFS envelope.
///
/// Two writers produce that shape, and nothing in the bytes tells them apart:
/// MinIO, whose data key is sealed under the static master key, and a RustFS
/// `rio-v2` build on a backend that persisted its native ciphertext bare.
/// MinIO's sealing is authenticated, so opening it under the local master key
/// is a proof rather than a guess: the bare blob is offered to the KMS only
/// once that has failed, or when no local master key exists to try. Without a
/// backend that reads bare ciphertext the MinIO route is the only one, exactly
/// as before.
#[cfg(feature = "rio-v2")]
pub(crate) async fn decrypt_unenveloped_sealed_dek(
bare_kms: Option<Arc<dyn SseDekProvider + '_>>,
local: Result<Arc<dyn SseDekProvider>, SseError>,
encrypted_dek: &[u8],
kms_key_id: &str,
context: &ObjectEncryptionContext,
) -> Result<[u8; 32], SseError> {
let Some(kms) = bare_kms else {
return local?.decrypt_minio_sse_dek(encrypted_dek, kms_key_id, context).await;
};
if let Ok(local) = local
&& let Ok(data_key) = local.decrypt_minio_sse_dek(encrypted_dek, kms_key_id, context).await
{
return Ok(data_key);
}
kms.decrypt_sse_dek(encrypted_dek, kms_key_id, context).await
}
/// The provider that unwraps KMS-wrapped data keys.
///
/// A provider registered through `set_sse_dek_provider_for_test` stands in for
/// the fresh [`KmsSseDekProvider`], which cannot resolve a KMS service from a
/// host that has none. Only `GLOBAL_KMS_DEK_PROVIDER` is consulted, never
/// `GLOBAL_SSE_DEK_PROVIDER`, so a local provider cached by an earlier
/// `get_local_sse_dek_provider` call cannot be selected to unwrap a KMS envelope.
pub(crate) async fn kms_sse_dek_provider<'a>(host: &'a dyn SseHost) -> Result<Arc<dyn SseDekProvider + 'a>, SseError> {
if let Some(cached) = GLOBAL_KMS_DEK_PROVIDER.read().ok().and_then(|guard| guard.as_ref().cloned()) {
return Ok(cached);
}
Ok(Arc::new(KmsSseDekProvider::new(host).await?))
}
/// The KMS provider, when its backend may own a stored data key that is not a
/// RustFS envelope. `None` without a running KMS, which leaves such a key to
/// the local provider exactly as before.
pub(crate) async fn bare_data_key_kms_provider<'a>(host: &'a dyn SseHost) -> Option<Arc<dyn SseDekProvider + 'a>> {
let provider = kms_sse_dek_provider(host).await.ok()?;
provider.reads_bare_data_key_ciphertext().await.then_some(provider)
}
/// Global SSE DEK provider cache for local / test providers.
///
/// Populated by `get_local_sse_dek_provider` and (for backward-compat in tests)
/// `set_sse_dek_provider_for_test`. Read by `get_local_sse_dek_provider` only —
/// the KMS-envelope decrypt path uses `GLOBAL_KMS_DEK_PROVIDER` so that a cached
/// local provider can never be selected for a KMS-wrapped data-key.
pub(crate) static GLOBAL_SSE_DEK_PROVIDER: LazyLock<RwLock<Option<Arc<dyn SseDekProvider>>>> =
LazyLock::new(|| RwLock::new(None));
/// Global KMS DEK provider cache for test-injected KMS providers.
///
/// Populated **only** by `set_sse_dek_provider_for_test` (test-only).
/// Read **only** by the KMS-envelope branch of `apply_managed_decryption_material`.
/// Separation from `GLOBAL_SSE_DEK_PROVIDER` prevents a previously-cached local
/// provider from being selected to unwrap a KMS data-key envelope.
static GLOBAL_KMS_DEK_PROVIDER: LazyLock<RwLock<Option<Arc<dyn SseDekProvider>>>> = LazyLock::new(|| RwLock::new(None));
/// Get or initialize the global SSE DEK provider
///
/// Factory function that automatically selects the appropriate provider:
/// - If `__RUSTFS_SSE_SIMPLE_CMK` environment variable exists: use SimpleSseDekProvider (test mode)
/// - Otherwise: use KmsSseDekProvider (production mode with real KMS)
///
/// # Returns
/// Arc to the global SSE DEK provider instance
///
/// # Example
/// ```rust,ignore
/// let provider = get_sse_dek_provider(host).await?;
/// let (data_key, encrypted_dek) = provider
/// .generate_sse_dek("bucket", "key", "kms-key-id")
/// .await?;
/// ```
pub(crate) async fn get_sse_dek_provider<'a>(host: &'a dyn SseHost) -> Result<Arc<dyn SseDekProvider + 'a>, SseError> {
if host.encryption_service().await.is_some() {
debug!("Using KmsSseDekProvider (KMS configured)");
return Ok(Arc::new(KmsSseDekProvider::new(host).await?));
}
get_local_sse_dek_provider()
}
/// Get the local data-key provider, loading and caching it on first use.
///
/// Startup calls this to check the local master key ahead of traffic; a request
/// that arrives without that step loads it the same way and gets the same error.
pub(crate) fn get_local_sse_dek_provider() -> Result<Arc<dyn SseDekProvider>, SseError> {
// An explicitly injected test provider overrides both the cache and the
// environment-based selection below.
#[cfg(any(test, feature = "test-util"))]
if let Some(injected) = test_injected_sse_dek_provider() {
return Ok(injected);
}
// Check if already initialized
if let Some(provider) = GLOBAL_SSE_DEK_PROVIDER
.read()
.map_err(|_| SseError::internal("Failed to read global SSE DEK provider cache"))?
.as_ref()
.cloned()
{
return Ok(provider);
}
// In test mode, prefer the simple CMK provider when the env var is set.
#[cfg(any(test, feature = "test-util"))]
{
if std::env::var("__RUSTFS_SSE_SIMPLE_CMK").is_ok() {
debug!("Using LocalSseDekProvider (test mode) based on __RUSTFS_SSE_SIMPLE_CMK");
let provider: Arc<dyn SseDekProvider> = Arc::new(LocalSseDekProvider::new()?);
let mut slot = GLOBAL_SSE_DEK_PROVIDER
.write()
.map_err(|_| SseError::internal("Failed to update global SSE DEK provider cache"))?;
if let Some(existing) = slot.as_ref() {
return Ok(existing.clone());
}
*slot = Some(provider.clone());
return Ok(provider);
}
}
// Production fallback: local SSE-S3 provider (no KMS configured).
debug!("Using local SSE-S3 provider (KMS not configured)");
let provider: Arc<dyn SseDekProvider> = Arc::new(LocalSseDekProvider::new_from_env()?);
let mut slot = GLOBAL_SSE_DEK_PROVIDER
.write()
.map_err(|_| SseError::internal("Failed to update global SSE DEK provider cache"))?;
if let Some(existing) = slot.as_ref() {
return Ok(existing.clone());
}
*slot = Some(provider.clone());
Ok(provider)
}
/// Reset both global SSE DEK provider caches (for testing only)
///
/// Clears GLOBAL_SSE_DEK_PROVIDER (local/test providers) and
/// GLOBAL_KMS_DEK_PROVIDER (test-injected KMS providers).
#[cfg(any(test, feature = "test-util"))]
pub fn reset_sse_dek_provider() {
if let Ok(mut slot) = GLOBAL_SSE_DEK_PROVIDER.write() {
*slot = None;
}
if let Ok(mut slot) = GLOBAL_KMS_DEK_PROVIDER.write() {
*slot = None;
}
}
#[cfg(any(test, feature = "test-util"))]
pub fn set_sse_dek_provider_for_test(provider: Arc<dyn SseDekProvider>) {
if let Ok(mut slot) = GLOBAL_KMS_DEK_PROVIDER.write() {
*slot = Some(provider.clone());
}
if let Ok(mut slot) = GLOBAL_SSE_DEK_PROVIDER.write() {
*slot = Some(provider);
}
}
/// Provider explicitly injected via `set_sse_dek_provider_for_test`, if any.
///
/// Reads `GLOBAL_KMS_DEK_PROVIDER` because that slot is populated *only* by the
/// test setter, so a hit here always means an explicit test injection.
#[cfg(any(test, feature = "test-util"))]
fn test_injected_sse_dek_provider() -> Option<Arc<dyn SseDekProvider>> {
GLOBAL_KMS_DEK_PROVIDER.read().ok().and_then(|guard| guard.as_ref().cloned())
}
-110
View File
@@ -1,110 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! S3 server-side encryption: which key encrypts an object, and what is stored to find it again.
//!
//! Covers the three S3 schemes:
//! - SSE-S3 (`AES256`): a data key wrapped by a key the server owns
//! - SSE-KMS (`aws:kms`): a data key wrapped by a KMS key the caller names
//! - SSE-C: a key the client supplies on every request and the server never stores
//!
//! ## Where this crate sits
//!
//! Between the S3 layer and the storage engine, depending on neither. It decides
//! the effective scheme of a write, obtains and unwraps data keys, authorizes the
//! use of a KMS key, and encodes and reads the encryption metadata of an object.
//! It does not transform object bytes: it hands back key material
//! ([`EncryptionMaterial`], [`DecryptionMaterial`]) and the caller applies it.
//!
//! What it needs from the running server — the KMS service, a bucket's default
//! encryption, a key-permission decision — arrives through [`SseHost`], passed
//! to every entry point that consults the server. Failures are reported as
//! [`SseError`], already classified with the S3 error code the caller answers with.
//!
//! ## Entry points
//!
//! - [`sse_encryption`] and [`sse_prepare_encryption`] resolve the material of a write;
//! [`encryption_material_to_metadata`] turns it into the metadata to store.
//! - [`sse_decryption`] resolves the material of a read from stored metadata;
//! [`classify_sse_read_response`] derives the response headers without unwrapping a key.
//! - [`rewrap_object_encryption_metadata`] moves a stored data key onto the current
//! version of its KMS key.
//! - [`startup::init`] settles the environment-driven configuration once per process
//! and reports it; calling it is optional.
//!
//! ```rust,ignore
//! let request = EncryptionRequest {
//! bucket: &bucket,
//! key: &key,
//! server_side_encryption: effective_sse,
//! ssekms_key_id: effective_kms_key_id,
//! ssekms_context: None,
//! sse_customer_algorithm: None,
//! sse_customer_key: None,
//! sse_customer_key_md5: None,
//! content_size: actual_size,
//! principal: principal.as_ref(),
//! };
//!
//! if let Some(material) = sse_encryption(host, request).await? {
//! metadata.extend(encryption_material_to_metadata(&material)?);
//! }
//! ```
mod core;
mod error;
mod host;
mod kms;
pub mod startup;
#[cfg(any(test, feature = "test-util"))]
pub mod testing;
#[cfg(test)]
#[allow(unused_imports)]
mod tests;
pub use crate::core::audit::KmsRequestAuditScope;
pub use crate::core::keying::material::{DecryptionMaterial, EncryptionKeyKind, EncryptionMaterial, ManagedSealedKey};
pub use crate::core::keying::metadata::{
ConflictingEncryptionMetadata, encryption_material_to_metadata, mark_encrypted_multipart_metadata,
normalize_encryption_metadata_case, stored_ssec_metadata, strip_managed_encryption_metadata,
};
pub use crate::core::read::{DecryptionRequest, classify_sse_read_response, sse_decryption};
pub use crate::core::request::SseKmsPrincipal;
#[cfg(any(test, feature = "test-util"))]
pub use crate::core::request::TestAuthorizationHooks;
pub use crate::core::rewrap::{ObjectDekRewrapOutcome, rewrap_object_encryption_metadata};
pub use crate::core::s3::authorization::{authorize_sse_kms_multipart_write, authorize_sse_kms_object_read};
pub use crate::core::s3::headers::{
SseReadResponseHeaders, SsecParamsFromHeaders, build_ssec_read_headers, extract_server_side_encryption_from_headers,
extract_ssec_params_from_headers, extract_ssekms_context_from_headers, project_sse_read_response_headers,
validate_sse_headers_for_read, validate_sse_headers_for_write,
};
pub use crate::core::s3::scheme::{DEFAULT_SSE_ALGORITHM, bucket_default_write_sse};
pub use crate::core::s3::ssec::{ssec_invalid_request, validate_ssec_for_read};
pub use crate::core::write::{EncryptionRequest, PrepareEncryptionRequest, sse_encryption, sse_prepare_encryption};
pub use crate::error::{KMS_KEY_NOT_FOUND_ERROR_CODE, SseError, data_plane_kms_error};
pub use crate::host::{BucketDefaultEncryption, BucketEncryption, KmsKeyAuthorizer, SseHost};
pub use crate::kms::local::LocalSseDekProvider;
#[cfg(any(test, feature = "test-util"))]
pub use crate::kms::local::TestSseDekProvider;
pub use crate::kms::provider::SseDekProvider;
#[cfg(any(test, feature = "test-util"))]
pub use crate::kms::provider::{reset_sse_dek_provider, set_sse_dek_provider_for_test};
// The managed-SSE classifier lives in the shared encryption-keys module so the
// scanner can reuse it; re-exported here so SSE callers find it beside the rest.
pub use rustfs_utils::http::object_encryption_keys::SSEType;
pub use rustfs_utils::http::object_encryption_keys::contains_managed_encryption_metadata;
pub(crate) const LOG_COMPONENT_STORAGE: &str = "storage";
pub(crate) const LOG_SUBSYSTEM_SSE: &str = "sse";
-250
View File
@@ -1,250 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The crate's process-startup step.
//!
//! [`init`] settles the two things that cannot change without a restart, both
//! read from the environment, and reports them once:
//!
//! - whether SSE-KMS requests are authorized against the key they name;
//! - whether the local SSE-S3 master key is usable.
//!
//! Nothing that follows the KMS service is decided here. The KMS can be
//! configured, started, restarted, and stopped while the server runs, so the
//! choice between the KMS and the local master key is made per request, and no
//! KMS state is cached by this crate.
//!
//! Calling [`init`] is optional. A process that never calls it reads the same
//! switch and loads the same master key on first use, with the same errors; it
//! only goes without the startup report.
use crate::error::SseError;
use crate::kms::local::{ENV_SSE_S3_MASTER_KEY, local_master_key_configured};
use crate::kms::provider::get_local_sse_dek_provider;
use crate::{LOG_COMPONENT_STORAGE, LOG_SUBSYSTEM_SSE};
use rustfs_config::{DEFAULT_KMS_ENFORCE_SSE_KEY_POLICY, ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY};
use rustfs_utils::get_env_bool;
use std::sync::{LazyLock, OnceLock};
const EVENT_SSE_KMS_KEY_POLICY_MODE: &str = "sse_kms_key_policy_mode";
const EVENT_SSE_LOCAL_MASTER_KEY: &str = "sse_local_master_key";
/// State of the local SSE-S3 master key at startup.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LocalMasterKeyStatus {
/// Configured and usable; SSE-S3 works without a KMS.
Loaded,
/// Not configured; SSE-S3 requires a running KMS.
NotSet,
/// Configured but unusable; SSE-S3 requests that need it are rejected.
Invalid,
}
impl LocalMasterKeyStatus {
fn as_str(self) -> &'static str {
match self {
Self::Loaded => "loaded",
Self::NotSet => "not_set",
Self::Invalid => "invalid",
}
}
}
/// What [`init`] settled for this process.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct StartupReport {
/// Whether SSE-KMS requests are authorized against the KMS key they name.
pub key_policy_enforced: bool,
/// State of the local SSE-S3 master key.
pub local_master_key: LocalMasterKeyStatus,
}
/// Settle the process-wide SSE configuration and report it.
///
/// Idempotent: the first call does the work and logs; later calls, such as a
/// second embedded server in the same process, return the same report silently.
/// An unusable master key does not fail startup, because a server whose KMS is
/// running never needs it.
pub fn init() -> StartupReport {
static REPORT: OnceLock<StartupReport> = OnceLock::new();
*REPORT.get_or_init(|| {
let key_policy_enforced = key_policy_enforced();
log_key_policy_mode(key_policy_enforced);
let local_master_key = load_local_master_key();
log_local_master_key(&local_master_key);
StartupReport {
key_policy_enforced,
local_master_key: local_master_key.unwrap_or(LocalMasterKeyStatus::Invalid),
}
})
}
/// Whether per-key KMS authorization is enforced on the SSE-KMS data path.
///
/// Frozen on first read, by [`init`] or by the first request, so one process
/// never answers the same question two ways.
pub(crate) fn key_policy_enforced() -> bool {
static ENFORCED: LazyLock<bool> =
LazyLock::new(|| get_env_bool(ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY, DEFAULT_KMS_ENFORCE_SSE_KEY_POLICY));
*ENFORCED
}
/// Load the local master key through the loader the request path uses.
///
/// An absent key is a state, not a failure: only a configured key is loaded, so
/// the error case is exactly "configured but unusable".
fn load_local_master_key() -> Result<LocalMasterKeyStatus, SseError> {
if !local_master_key_configured() {
return Ok(LocalMasterKeyStatus::NotSet);
}
get_local_sse_dek_provider().map(|_| LocalMasterKeyStatus::Loaded)
}
/// Report the configured SSE-KMS authorization mode.
///
/// The disabled case warns rather than logs: while enforcement is off, any identity
/// allowed to write an object can encrypt it under any key, and operators should hear
/// about that even though disabled is the long-term default.
fn log_key_policy_mode(enforced: bool) {
if enforced {
tracing::info!(
event = EVENT_SSE_KMS_KEY_POLICY_MODE,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
enforced = true,
"SSE-KMS requests are authorized against the resolved KMS key (kms:GenerateDataKey / kms:Decrypt)"
);
return;
}
tracing::warn!(
event = EVENT_SSE_KMS_KEY_POLICY_MODE,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
enforced = false,
"SSE-KMS requests are not authorized against the KMS key they name; any identity allowed to \
write an object may encrypt it under any key, and any identity allowed to read it may have it \
decrypted. Grant kms:GenerateDataKey and kms:Decrypt on the keys your workloads use, then set \
{ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY}=true."
);
}
/// Report the local master key's state.
///
/// The failure text is the one a request is answered with. It describes the
/// configuration problem and never carries the configured value.
fn log_local_master_key(outcome: &Result<LocalMasterKeyStatus, SseError>) {
match outcome {
Ok(status @ LocalMasterKeyStatus::Loaded) => tracing::info!(
event = EVENT_SSE_LOCAL_MASTER_KEY,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
state = status.as_str(),
"{ENV_SSE_S3_MASTER_KEY} is loaded; SSE-S3 uses it whenever no KMS is running"
),
Ok(status) => tracing::info!(
event = EVENT_SSE_LOCAL_MASTER_KEY,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
state = status.as_str(),
"{ENV_SSE_S3_MASTER_KEY} is not set; SSE-S3 requires a running KMS"
),
Err(error) => tracing::error!(
event = EVENT_SSE_LOCAL_MASTER_KEY,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_SSE,
state = LocalMasterKeyStatus::Invalid.as_str(),
reason = %error.message,
"{ENV_SSE_S3_MASTER_KEY} is set but unusable; SSE-S3 requests are rejected while no KMS is running"
),
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::kms::provider::reset_sse_dek_provider;
use crate::tests::lock_sse_test_state;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_s3_types::S3ErrorCode;
use temp_env::with_vars;
/// Run `check` with the local master key set to `value` and nothing cached or injected.
async fn with_master_key<T>(value: Option<&str>, check: impl FnOnce() -> T) -> T {
let _guard = lock_sse_test_state().await;
reset_sse_dek_provider();
let outcome = with_vars([(ENV_SSE_S3_MASTER_KEY, value), ("__RUSTFS_SSE_SIMPLE_CMK", None)], check);
reset_sse_dek_provider();
outcome
}
#[tokio::test]
async fn an_absent_master_key_is_reported_as_not_set() {
let status = with_master_key(None, load_local_master_key)
.await
.expect("an absent key is a state, not a failure");
assert_eq!(status, LocalMasterKeyStatus::NotSet);
let blank = with_master_key(Some(" "), load_local_master_key)
.await
.expect("a blank value counts as absent");
assert_eq!(blank, LocalMasterKeyStatus::NotSet);
}
#[tokio::test]
async fn a_valid_master_key_is_loaded_and_cached_for_requests() {
let encoded = BASE64_STANDARD.encode_to_string([7u8; 32]);
let cached = with_master_key(Some(&encoded), || {
let status = load_local_master_key().expect("a 32-byte key should load");
assert_eq!(status, LocalMasterKeyStatus::Loaded);
// The request path must find the provider startup loaded, not load a second one.
crate::kms::provider::GLOBAL_SSE_DEK_PROVIDER
.read()
.expect("provider cache lock")
.is_some()
})
.await;
assert!(cached, "startup should leave the provider cached for the request path");
}
#[tokio::test]
async fn an_unusable_master_key_fails_the_way_a_request_would() {
let short = BASE64_STANDARD.encode_to_string([7u8; 16]);
for unusable in ["not base64 at all!", short.as_str()] {
let error = with_master_key(Some(unusable), load_local_master_key)
.await
.expect_err("an unusable key must not load");
assert_eq!(error.code, S3ErrorCode::InvalidRequest);
assert!(
!error.message.contains(unusable),
"the failure text must not carry the configured value: {}",
error.message
);
}
}
#[tokio::test]
async fn init_reports_once_and_repeats_the_same_report() {
let encoded = BASE64_STANDARD.encode_to_string([9u8; 32]);
let (first, second) = with_master_key(Some(&encoded), || (init(), init())).await;
assert_eq!(first, second, "a second call must return the first report");
assert_eq!(first.key_policy_enforced, key_policy_enforced());
// The report is settled once per process, so this holds whether this test
// or another caller of `init` ran first.
assert_eq!(init(), first);
}
}
-70
View File
@@ -1,70 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Fixtures for tests of SSE call sites, in this crate and in its dependents.
use crate::{KmsKeyAuthorizer, SseError, SseKmsPrincipal};
use async_trait::async_trait;
use base64_simd::STANDARD as BASE64_STANDARD;
use rustfs_policy::policy::action::KmsAction;
use rustfs_utils::http::headers::AMZ_ENCRYPTION_KMS;
use rustfs_utils::http::object_encryption_keys::{
INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_IV_HEADER, INTERNAL_ENCRYPTION_KEY_HEADER,
INTERNAL_ENCRYPTION_KEY_ID_HEADER,
};
use std::collections::HashMap;
use std::sync::{Arc, Mutex, PoisonError};
/// An authorizer that answers every request the same way and records what it was asked.
pub struct RecordingKmsKeyAuthorizer {
allowed: bool,
calls: Mutex<Vec<(KmsAction, String)>>,
}
impl RecordingKmsKeyAuthorizer {
pub fn new(allowed: bool) -> Arc<Self> {
Arc::new(Self {
allowed,
calls: Mutex::new(Vec::new()),
})
}
/// The `(action, key id)` pairs this authorizer was asked about, in order.
pub fn calls(&self) -> Vec<(KmsAction, String)> {
self.calls.lock().unwrap_or_else(PoisonError::into_inner).clone()
}
}
#[async_trait]
impl KmsKeyAuthorizer for RecordingKmsKeyAuthorizer {
async fn is_allowed(&self, _principal: &SseKmsPrincipal, action: KmsAction, key_id: &str) -> Result<bool, SseError> {
self.calls
.lock()
.unwrap_or_else(PoisonError::into_inner)
.push((action, key_id.to_string()));
Ok(self.allowed)
}
}
/// Stored metadata of an SSE-KMS object wrapped under the key `finance-key`.
pub fn sse_kms_object_metadata() -> HashMap<String, String> {
HashMap::from([
("x-amz-server-side-encryption".to_string(), AMZ_ENCRYPTION_KMS.to_string()),
("x-amz-server-side-encryption-aws-kms-key-id".to_string(), "finance-key".to_string()),
(INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), "finance-key".to_string()),
(INTERNAL_ENCRYPTION_KEY_HEADER.to_string(), BASE64_STANDARD.encode_to_string([7u8; 48])),
(INTERNAL_ENCRYPTION_IV_HEADER.to_string(), BASE64_STANDARD.encode_to_string([9u8; 12])),
(INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "aws:kms".to_string()),
])
}
File diff suppressed because it is too large Load Diff
-2
View File
@@ -30,11 +30,9 @@ Contract crates stay below implementation crates. Forbidden edges:
| `security-governance -> rustfs` | Governance contracts stay below the binary crate |
| `extension-schema -> rustfs` | The extension schema is consumed by the binary, never the reverse |
| `extension-schema -> ecstore` | The extension schema must not reach storage internals |
| `sse -> ecstore`, `sse -> iam`, `sse -> rustfs` | SSE key resolution reaches the running server only through its `SseHost` port |
- `rustfs-storage-api` exposes storage-facing replication status/state contracts only through `crates/storage-api/src/replication.rs`, so its temporary dependency on `rustfs-filemeta` wire types stays centralized and no `rustfs-replication` / `rustfs-storage-api` cycle appears.
- Leaf crates (`config`, `credentials`, `crypto`, `io-metrics`, `madmin`) may not depend on other `rustfs-*` crates, in either TOML spelling, except the adjudicated edges pinned in the guard's leaf allowlist: `io-metrics -> rustfs-s3-ops` (pure contract crates sharing the `S3Operation` vocabulary) and `madmin -> rustfs-signer` (the SigV4-signed admin SDK client). A new leaf exception must be a pure contract dependency (types and enums only, no I/O, no globals, no non-contract internal dependencies) and land together with its allowlist entry.
- `rustfs-sse` sits between the S3 layer and the storage engine and depends on neither. Its internal dependencies are pinned in the same guard to `rustfs-config`, `rustfs-kms`, `rustfs-policy`, `rustfs-s3-types`, and `rustfs-utils`; the KMS service, a bucket's default encryption, and key-permission decisions arrive through `rustfs_sse::SseHost`, which `rustfs/src/storage/sse.rs` implements. That file also owns every translation across the engine boundary (`ObjectEncryptionResolver`, `WriteEncryption`, `MultipartCompletionAuthorization`), so neither side names the other's types.
- Compile-time source reads follow the same direction: `include_str!` / `include!` of a `.rs` file must not resolve outside the including crate (`scripts/check_layer_dependencies.sh`). Shared source-text expectations belong in a contract surface such as `rustfs_protos::compat_manifest` (`crates/protos/src/compat_manifest.rs`) and are asserted by each owning crate.
## ECStore Access Boundary
+1 -1
View File
@@ -47,7 +47,7 @@ RustFS-side statics that matter architecturally because other modules are tempte
| `KEYSTONE_AUTH`, `KEYSTONE_MAPPER`, `KEYSTONE_CONFIG` | `rustfs/src/auth_keystone.rs` | Keystone provider, mapper, and config stay private to the Keystone owner. |
| `DEADLOCK_DETECTOR` | `rustfs/src/storage/deadlock_detector.rs` | Detector lifecycle stays private to the storage deadlock detector. |
| `CONCURRENCY_MANAGER` | `rustfs/src/storage/concurrency/manager.rs` | Storage concurrency scheduler state stays inside the concurrency owner. |
| `GLOBAL_KMS_DEK_PROVIDER`, `GLOBAL_SSE_DEK_PROVIDER` | `crates/sse/src/kms/provider.rs` | DEK provider caches stay private to the SSE owner. |
| `GLOBAL_KMS_DEK_PROVIDER`, `GLOBAL_SSE_DEK_PROVIDER` | `rustfs/src/storage/sse.rs` | DEK provider caches stay private to the SSE owner. |
| `ECSTORE_EVENT_DISPATCH_HOOK` | `rustfs/src/server/event.rs` | Event bridge registration goes through the storage facade. |
| `AUDIT_MODULE_ENABLED`, `NOTIFY_MODULE_ENABLED` | `rustfs/src/module_switches.rs` | Module toggles are read through module-switch helpers; `MODULE_SWITCH_RMW_LOCK` (`rustfs/src/server/module_switch.rs`) serializes persisted updates. |
| `RUNTIME_CONFIG_RELOAD_MUTEX` | `rustfs/src/admin/service/config.rs` | Serializes dynamic config reload fanout. |
+3 -3
View File
@@ -1,7 +1,7 @@
# KMS Bulk Rekey Job Contract
**Use this when:** changing the bulk envelope re-wrap sweep (`rustfs/src/kms_rekey.rs`), its admin endpoints (`rustfs/src/admin/handlers/kms_rekey.rs`), the re-wrap primitive, or anything that decides which objects a rekey may touch.
**Source of truth:** `rustfs/src/kms_rekey.rs`, `rustfs/src/admin/handlers/kms_rekey.rs`, `rewrap_object_encryption_metadata` in `crates/sse/src/core/rewrap.rs`, `KmsManager::rewrap_data_key` / `KmsManager::describe_data_key_wrapping` in `crates/kms/src/manager.rs`, `put_object_metadata` in `crates/ecstore/src/set_disk/ops/object.rs`.
**Source of truth:** `rustfs/src/kms_rekey.rs`, `rustfs/src/admin/handlers/kms_rekey.rs`, `rewrap_object_encryption_metadata` in `rustfs/src/storage/sse.rs`, `KmsManager::rewrap_data_key` / `KmsManager::describe_data_key_wrapping` in `crates/kms/src/manager.rs`, `put_object_metadata` in `crates/ecstore/src/set_disk/ops/object.rs`.
The bulk rekey job re-wraps stored data-key envelopes under the current key-encryption key (KEK) without rewriting object bodies. This document is the acceptance bar; where the shipped v1 sweep deliberately narrows it, [Implementation Status](#implementation-status-v1-sweep) records the deviation.
@@ -52,7 +52,7 @@ Idempotency comes from object metadata itself: the envelope's state is the targe
### Reading the wrapping KEK version
There is no key-version metadata key. Object metadata carries the key id (`x-rustfs-encryption-key-id`) and the sealed blob under `x-rustfs-encryption-key`; `DecryptResponse` in `crates/kms/src/types.rs` does not report a version either. The version is recoverable because the sealed blob is structured: for every backend that builds one, the ciphertext is the JSON of `DataKeyEnvelope` (`crates/kms/src/encryption/dek.rs`), and the read path already discriminates on it via `is_data_key_envelope` in `crates/sse/src/core/read.rs`.
There is no key-version metadata key. Object metadata carries the key id (`x-rustfs-encryption-key-id`) and the sealed blob under `x-rustfs-encryption-key`; `DecryptResponse` in `crates/kms/src/types.rs` does not report a version either. The version is recoverable because the sealed blob is structured: for every backend that builds one, the ciphertext is the JSON of `DataKeyEnvelope` (`crates/kms/src/encryption/dek.rs`), and the read path already discriminates on it via `is_data_key_envelope` in `rustfs/src/storage/sse.rs`.
| Backend | Rotates | Where the wrapping version lives | Recoverable by a scan |
|---|---|---|---|
@@ -95,7 +95,7 @@ Replication destinations are unresolved: propagation depends on `rustfs/backlog#
Three properties of `put_object_metadata` (`crates/ecstore/src/set_disk/ops/object.rs`) constrain the re-wrap write:
- **The merge is additive; it cannot remove keys.** Overwriting a key that keeps its name is safe; a re-wrap that changes *which* keys describe the envelope leaves the old keys behind. This is a live hazard: `parse_minio_managed_sealed_key` in `crates/sse/src/core/keying/object.rs` selects the MinIO decrypt branch on the mere presence of the MinIO seal-algorithm header, so a RustFS-native envelope written onto MinIO-compatible headers without neutralizing them steers reads down the stale branch. Any envelope-shape change must neutralize superseded keys in the same write.
- **The merge is additive; it cannot remove keys.** Overwriting a key that keeps its name is safe; a re-wrap that changes *which* keys describe the envelope leaves the old keys behind. This is a live hazard: `parse_minio_managed_sealed_key` in `rustfs/src/storage/sse.rs` selects the MinIO decrypt branch on the mere presence of the MinIO seal-algorithm header, so a RustFS-native envelope written onto MinIO-compatible headers without neutralizing them steers reads down the stale branch. Any envelope-shape change must neutralize superseded keys in the same write.
- **Object-lock retention is enforced before the merge.** `check_object_lock_retention_update` (`crates/ecstore/src/set_disk/mod.rs`) runs first; rekey inherits its decision and must not acquire a bypass.
- **`mod_time` is preserved unless the caller sets it.** The re-wrap path leaves it unset so age-based lifecycle rules are not perturbed.
@@ -1,13 +1,13 @@
# MinIO On-Disk Format Interoperability
**Use this when:** deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching any constant or codec listed under Version Anchors.
**Source of truth:** `crates/filemeta/src/filemeta.rs`, `crates/filemeta/src/filemeta/codec.rs`, `crates/ecstore/src/bucket/metadata.rs`, `crates/ecstore/src/bucket/migration.rs`, `crates/sse/src/core/`, `crates/sse/src/kms/`, `rustfs/Cargo.toml` `[features]`, `.github/workflows/ci.yml`, `.github/workflows/minio-interop.yml`.
**Source of truth:** `crates/filemeta/src/filemeta.rs`, `crates/filemeta/src/filemeta/codec.rs`, `crates/ecstore/src/bucket/metadata.rs`, `crates/ecstore/src/bucket/migration.rs`, `rustfs/src/storage/sse.rs`, `rustfs/Cargo.toml` `[features]`, `.github/workflows/ci.yml`, `.github/workflows/minio-interop.yml`.
This is an interop contract, not a plan. Migration is one-way (MinIO to RustFS). Erasure-coding internals are owned by [erasure-coding.md](erasure-coding.md); this document owns the interop claim, the fixture evidence, and the out-of-scope list.
## Scope Matrix By Build Variant
Build variants are the `rustfs` crate features in `rustfs/Cargo.toml`: `default`, `full`, and `rio-v2` (which enables `rustfs-ecstore/rio-v2` and `rustfs-sse/rio-v2`, and pulls in `crates/rio-v2`). `rio-v2` is absent from both `default` and `full`.
Build variants are the `rustfs` crate features in `rustfs/Cargo.toml`: `default`, `full`, and `rio-v2` (which enables `rustfs-ecstore/rio-v2` and pulls in `crates/rio-v2`). `rio-v2` is absent from both `default` and `full`.
| MinIO artifact | `default` / `full` build | `rio-v2` build | Notes |
|---|:--:|:--:|---|
@@ -81,7 +81,7 @@ The `xl.meta` around a MinIO SSE object parses in every build, so such objects l
### Seams
The cryptography (DARE v2 stream format, object-key derivation, sealing) was never the gap. Three metadata seams above it rejected MinIO objects; all are closed in `rio-v2` builds. Symbols are under `crates/sse/src/core/` unless noted.
The cryptography (DARE v2 stream format, object-key derivation, sealing) was never the gap. Three metadata seams above it rejected MinIO objects; all are closed in `rio-v2` builds. Symbols are in `rustfs/src/storage/sse.rs` unless noted.
| Seam | Resolution |
|---|---|
@@ -113,10 +113,10 @@ Inventory the source first: bucket default encryption means objects can be encry
| Fact | Value |
|---|---|
| Shipping status | Ships in no default build: absent from `default` and `full` in `rustfs/Cargo.toml`; released binaries and container images never include it. Enable with `--features rio-v2`. |
| Pull-request coverage | `test-and-lint-rio-v2` in `.github/workflows/ci.yml`: clippy plus `cargo nextest` for `rustfs`, `rustfs-ecstore`, and `rustfs-sse` with `--features rio-v2`. This is the cfg-seam guard; it keeps the feature compiling and its unit suite green on every pull request. |
| Pull-request coverage | `test-and-lint-rio-v2` in `.github/workflows/ci.yml`: clippy plus `cargo nextest` for `rustfs` and `rustfs-ecstore` with `--features rio-v2`. This is the cfg-seam guard; it keeps the feature compiling and its unit suite green on every pull request. |
| Full-suite lane | `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` in `ci.yml` run only on the nightly schedule and manual dispatch (`cache-warm.yml` keeps the `ci-feat-rio` cache warm so the scheduled build fits its timeout). |
| Interop evidence | `.github/workflows/minio-interop.yml` (nightly plus manual) regenerates real MinIO backend trees via `crates/rio-v2/tests/minio_fixture_lab/` and runs the `#[ignore]` reader tests in `rustfs/src/storage/minio_generated_read_test.rs` with `--features rio-v2`. Its freshness is tracked in `.github/scheduled-validations.json`. |
| Promote-or-delete condition | The variant stays dormant until one of two things happens. **Promote**: a release commits to shipping MinIO SSE migration as a supported capability; then `rio-v2` joins `default`/`full`, the scheduled lanes run on every pull request, and this section is rewritten. **Delete**: no release commits to it and the scheduled lanes are not kept green; then the feature flag, `crates/rio-v2`, the cfg seams under `crates/sse/src/`, the three `ci.yml` jobs, the `cache-warm.yml` warm step, `minio-interop.yml`, and its `scheduled-validations.json` entry are removed in one change. Either outcome must update `ARCHITECTURE.md` and the `ci.yml` job comments that cite this section. |
| Promote-or-delete condition | The variant stays dormant until one of two things happens. **Promote**: a release commits to shipping MinIO SSE migration as a supported capability; then `rio-v2` joins `default`/`full`, the scheduled lanes run on every pull request, and this section is rewritten. **Delete**: no release commits to it and the scheduled lanes are not kept green; then the feature flag, `crates/rio-v2`, the cfg seams in `rustfs/src/storage/sse.rs`, the three `ci.yml` jobs, the `cache-warm.yml` warm step, `minio-interop.yml`, and its `scheduled-validations.json` entry are removed in one change. Either outcome must update `ARCHITECTURE.md` and the `ci.yml` job comments that cite this section. |
## Fixture Evidence
+1 -1
View File
@@ -1,7 +1,7 @@
# AWS KMS backend: setup, verification, and operations
**Use this when:** connecting RustFS to AWS KMS for the first time, verifying an existing AWS-backed deployment, upgrading one from 1.0.4 or 1.0.5, or deciding how to run and protect the AWS keys RustFS depends on.
**Source of truth:** `crates/kms/src/backends/aws.rs` (backend behavior and its deviations), `crates/kms/src/config.rs` (`ENV_KMS_AWS_*`), `crates/kms/src/policy.rs` (retry, deadline and circuit breaker), `crates/sse/src/core/` (SSE data path). 中文版: [kms-aws-backend-guide.zh-CN.md](kms-aws-backend-guide.zh-CN.md).
**Source of truth:** `crates/kms/src/backends/aws.rs` (backend behavior and its deviations), `crates/kms/src/config.rs` (`ENV_KMS_AWS_*`), `crates/kms/src/policy.rs` (retry, deadline and circuit breaker), `rustfs/src/storage/sse.rs` (SSE data path). 中文版: [kms-aws-backend-guide.zh-CN.md](kms-aws-backend-guide.zh-CN.md).
The statements about behavior on this page were observed against the real AWS KMS service (region `eu-north-1`) and, for failures AWS cannot be asked to produce on demand, against a loopback stand-in that speaks the AWS KMS wire protocol. Where a statement is guidance that was not exercised, it says so. For the contract-level differences between AWS and the other backends, see [KMS backend security properties](kms-backend-security.md#aws-kms-deviations-from-the-shared-backend-contract).
@@ -1,7 +1,7 @@
# AWS KMS 后端:接入、验证与运维
**适用场景:** 首次把 RustFS 接入 AWS KMS、验证已有的 AWS 后端部署、从 1.0.4 或 1.0.5 升级,或者规划如何运维并保护 RustFS 所依赖的 AWS 密钥。
**事实来源:** `crates/kms/src/backends/aws.rs`(后端行为及其差异)、`crates/kms/src/config.rs`(`ENV_KMS_AWS_*`)、`crates/kms/src/policy.rs`(重试、超时与熔断)、`crates/sse/src/core/`(SSE 数据面)。English: [kms-aws-backend-guide.md](kms-aws-backend-guide.md)。
**事实来源:** `crates/kms/src/backends/aws.rs`(后端行为及其差异)、`crates/kms/src/config.rs`(`ENV_KMS_AWS_*`)、`crates/kms/src/policy.rs`(重试、超时与熔断)、`rustfs/src/storage/sse.rs`(SSE 数据面)。English: [kms-aws-backend-guide.md](kms-aws-backend-guide.md)。
本页关于行为的描述,均来自对真实 AWS KMS 服务(区域 `eu-north-1`)的实测;对于无法让 AWS 按需产生的故障,则使用一个实现了 AWS KMS 线上协议的本机替身来实测。属于未经实测的建议之处,文中会明确标注。AWS 与其它后端在契约层面的差异,见 [KMS backend security properties](kms-backend-security.md#aws-kms-deviations-from-the-shared-backend-contract)。
Generated
+6 -6
View File
@@ -2,11 +2,11 @@
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1794495034,
"narHash": "sha256-E0hYoioqUtM2XmPPvYYpJgdADF5AW9oOoPSEiM9dTHQ=",
"lastModified": 1795249095,
"narHash": "sha256-8JDgyVvWUZg5m9hNlL8Ge9aT647ng3Ukc8NPMiKNTPQ=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "2074cc672349f14240fb7ea90d509e0b643e9dd4",
"rev": "93a9ed0a866099a230ed5146a5bb7d16fd5c9b14",
"type": "github"
},
"original": {
@@ -29,11 +29,11 @@
]
},
"locked": {
"lastModified": 1794646145,
"narHash": "sha256-rpqDyUM/d5fACYB8W+wKxxO57ICWjjalzb2X25oFFkY=",
"lastModified": 1795253373,
"narHash": "sha256-Vg6Sg8gJCDG9qpVEdOZX+72Z50MLwdnwtfZ8FNBwiYI=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "5b87ee0513c29127a01ad06e9cb362ea5ce91e78",
"rev": "52db20539ec0ba063479d66092992d9efcf456ff",
"type": "github"
},
"original": {
+1 -3
View File
@@ -80,7 +80,7 @@ e2e-test-hooks = ["rustfs-ecstore/e2e-test-hooks", "rustfs-scanner/e2e-test-hook
connect-e2e-short-credentials = []
# Builds the dedicated rustfs-cli-e2e target with a build-time public enrollment root.
offline-enrollment-e2e-root = []
rio-v2 = ["rustfs-ecstore/rio-v2", "rustfs-sse/rio-v2"]
rio-v2 = ["rustfs-ecstore/rio-v2"]
# Linux-only cuObject S3-over-RDMA GET into client (GPU) memory; needs libcuobjserver.
cuobject = ["dep:rustfs-cuobject", "rustfs-cuobject/server"]
# NIXL/UCX transport for S3-over-RDMA; replaces cuobject when both are enabled.
@@ -281,7 +281,6 @@ rustfs-rio = { workspace = true }
rustfs-s3-client = { workspace = true }
rustfs-s3-types = { workspace = true, features = ["compat-s3s"] }
rustfs-s3-ops = { workspace = true }
rustfs-sse = { workspace = true }
rustfs-security-governance = { workspace = true }
rustfs-data-usage = { workspace = true }
rustfs-s3select-api = { workspace = true }
@@ -425,7 +424,6 @@ h2.workspace = true
rustfs-targets = { workspace = true, features = ["test-support"] }
uuid = { workspace = true, features = ["v4", "v5", "fast-rng", "macro-diagnostics"] }
rustfs-iam = { workspace = true, features = ["test-util"] }
rustfs-sse = { workspace = true, features = ["test-util"] }
serial_test = { workspace = true }
tempfile = { workspace = true }
aws-config = { workspace = true }
+1 -1
View File
@@ -62,7 +62,7 @@ use super::storage_api::multipart_usecase::s3_api::multipart::{
};
use super::storage_api::multipart_usecase::set_disk::is_valid_storage_class;
use super::storage_api::multipart_usecase::sse::{
DecryptionRequest, EncryptionKeyKind, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal, SseKmsPrincipalExt,
DecryptionRequest, EncryptionKeyKind, EncryptionRequest, PrepareEncryptionRequest, SseKmsPrincipal,
apply_bucket_default_lock_retention, authorize_sse_kms_multipart_write, authorize_sse_kms_object_read,
build_ssec_read_headers, encryption_material_to_metadata, extract_server_side_encryption_from_headers,
extract_ssec_params_from_headers, extract_ssekms_context_from_headers, get_buffer_size_opt_in,
+16 -199
View File
@@ -1082,8 +1082,6 @@ mod tests {
#[tokio::test]
#[serial_test::serial]
async fn compressed_complete_records_logical_quota_usage_and_overwrite_delta() {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("compressed-complete-quota", 16_384).await;
let object = "object";
@@ -1123,53 +1121,10 @@ mod tests {
}))
.build()
.expect("complete multipart input should build");
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let completed = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None))
.await
}
}
}
};
completed.expect("compressed multipart completion should succeed");
usecase
.complete_for_s3s_request(build_request(input, Method::POST))
.await
.expect("compressed multipart completion should succeed");
let quota = quota_checker
.check_quota(&bucket, QuotaOperation::PutObject, 0)
@@ -2228,55 +2183,10 @@ mod tests {
.multipart_upload(Some(CompletedMultipartUpload { parts: Some(Vec::new()) }))
.build()
.expect("complete multipart input should build");
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let usecase = DefaultMultipartUsecase::from_global();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let rejected = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None)).await
}
}
}
};
let err = rejected.expect_err("empty parts list must be rejected");
let err = DefaultMultipartUsecase::from_global()
.complete_for_s3s_request(build_request(input, Method::POST))
.await
.expect_err("empty parts list must be rejected");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
let current_info = store
@@ -2308,53 +2218,10 @@ mod tests {
}))
.build()
.expect("complete multipart input should build");
let usecase = DefaultMultipartUsecase::from_global();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let completed = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(owned_envelope(Method::POST), owned, None)).await
}
}
}
};
let completed = completed.expect("staged part should remain completable after empty request rejection");
let completed = DefaultMultipartUsecase::from_global()
.complete_for_s3s_request(build_request(input, Method::POST))
.await
.expect("staged part should remain completable after empty request rejection");
assert!(completed.e_tag.is_some());
let completed_info = store
.get_object_info(
@@ -2696,60 +2563,10 @@ mod tests {
}))
.build()
.expect("complete input");
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let usecase = DefaultMultipartUsecase::from_global();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let rejected = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => Err(crate::app::dto_values::seam_error(error)),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(usecase.execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(usecase.execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
}
}
};
let error = rejected.expect_err("absent upload cannot complete");
let error = DefaultMultipartUsecase::from_global()
.complete_for_s3s_request(build_request(input, Method::POST))
.await
.expect_err("absent upload cannot complete");
assert_eq!(probe.audits.lock().expect("audits").len(), 0, "{shape}/{label}");
assert_eq!(probe.events.lock().expect("events").len(), 0, "{shape}/{label}");
let (code, status, message, storage_cause) = if raw_id.is_some() {
@@ -382,8 +382,7 @@ impl DefaultMultipartUsecase {
content_size: actual_size,
principal: None,
}
.validate_multipart_ssec(&fi.user_defined)
.map_err(ApiError::from)?;
.validate_multipart_ssec(&fi.user_defined)?;
}
let (requested_sse, requested_kms_key_id) = if has_ssec {
let ssec_material = sse_decryption(DecryptionRequest {
@@ -313,8 +313,7 @@ impl DefaultMultipartUsecase {
content_size: 0,
principal: None,
}
.validate_multipart_ssec(&mp_info.user_defined)
.map_err(ApiError::from)?;
.validate_multipart_ssec(&mp_info.user_defined)?;
let src_versioned = BucketVersioningSys::prefix_enabled(&src_bucket, &src_key).await;
let src_version_suspended = BucketVersioningSys::prefix_suspended(&src_bucket, &src_key).await;
+5 -5
View File
@@ -108,11 +108,11 @@ use super::storage_api::object_usecase::set_disk::{
get_lock_acquire_timeout, get_object_disk_read_timeout, is_valid_storage_class,
};
use super::storage_api::object_usecase::sse::{
DecryptionRequest, EncryptionMaterialExt, EncryptionRequest, SseKmsPrincipal, SseKmsPrincipalExt,
apply_bucket_default_lock_retention, authorize_sse_kms_object_read, bucket_default_write_sse, build_ssec_read_headers,
classify_sse_read_response, encryption_material_to_metadata, extract_server_side_encryption_from_headers,
extract_ssec_params_from_headers, extract_ssekms_context_from_headers, get_buffer_size_opt_in,
load_bucket_object_lock_config_state, map_get_object_reader_error, sse_encryption, validate_bucket_object_lock_enabled_state,
DecryptionRequest, EncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention, authorize_sse_kms_object_read,
bucket_default_write_sse, build_ssec_read_headers, classify_sse_read_response, encryption_material_to_metadata,
extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, extract_ssekms_context_from_headers,
get_buffer_size_opt_in, load_bucket_object_lock_config_state, map_get_object_reader_error, sse_encryption,
validate_bucket_object_lock_enabled_state,
};
use super::storage_api::object_usecase::storage_class as storageclass;
use super::storage_api::object_usecase::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
+1 -3
View File
@@ -2,9 +2,7 @@
use super::storage_api::select_object::StorageError;
use super::storage_api::select_object::options::get_opts;
use super::storage_api::select_object::request_context::spawn_traced;
use super::storage_api::select_object::sse::{
SseKmsPrincipal, SseKmsPrincipalExt, authorize_sse_kms_object_read, project_sse_read_response_headers,
};
use super::storage_api::select_object::sse::{SseKmsPrincipal, authorize_sse_kms_object_read, project_sse_read_response_headers};
use super::storage_api::select_object::{
StoragePrepareSelectObjectSnapshotError, StorageSelectObjectSnapshot, get_validated_store, validate_sse_headers_for_read,
validate_ssec_for_read,
+3 -3
View File
@@ -1675,9 +1675,9 @@ pub(crate) mod sse {
validate_bucket_object_lock_enabled_state,
};
pub(crate) use crate::storage::storage_api::sse_consumer::{
EncryptionKeyKind, EncryptionMaterialExt, SseKmsPrincipalExt, bucket_default_write_sse, build_ssec_read_headers,
encryption_material_to_metadata, extract_ssec_params_from_headers, extract_ssekms_context_from_headers,
map_get_object_reader_error, mark_encrypted_multipart_metadata, multipart_completion_authorization, stored_ssec_metadata,
EncryptionKeyKind, bucket_default_write_sse, build_ssec_read_headers, encryption_material_to_metadata,
extract_ssec_params_from_headers, extract_ssekms_context_from_headers, map_get_object_reader_error,
mark_encrypted_multipart_metadata, multipart_completion_authorization, stored_ssec_metadata,
};
}
+72 -13
View File
@@ -18,7 +18,6 @@ use http::StatusCode;
use rustfs_kms::KmsUnavailableError;
use rustfs_lock::LockError;
use rustfs_s3_types::S3ErrorCode;
use rustfs_sse::{SseError, data_plane_kms_error};
use s3s::S3Error as LegacyS3Error;
use s3s::stream::{aws_chunked_stream::AwsChunkedStreamError, upload_stream::UploadStreamError};
@@ -52,7 +51,7 @@ pub(crate) fn rpc_replay_admission_exhausted_status() -> tonic::Status {
const UNSUPPORTED_PARTICIPANT_MESSAGE: &str = "A storage participant does not support this operation's protocol version. If the cluster is being upgraded, finish upgrading every node and retry.";
/// S3 error code for a request that names a KMS key the KMS does not hold.
pub use rustfs_sse::KMS_KEY_NOT_FOUND_ERROR_CODE;
pub const KMS_KEY_NOT_FOUND_ERROR_CODE: &str = "KMS.NotFoundException";
fn lock_resource_class(resource: Option<&str>) -> &'static str {
let Some(resource) = resource else {
@@ -123,6 +122,77 @@ fn record_s3_lock_failure(err: &StorageError) {
}
}
/// Map a KMS failure that surfaced on the S3 data path to its S3 error code.
///
/// The contract deliberately differs from the admin lifecycle handlers
/// (`kms_key_lifecycle::lifecycle_error_status`): there a key id is the
/// resource being addressed, so a missing key is `404`. Here the key id
/// arrives inside a request header or a bucket default, so a key that is
/// missing, disabled, or otherwise unusable is configuration the caller has to
/// correct — AWS answers `400`, and reporting `500` instead both misfiles the
/// failure as a server fault and makes SDKs back off and retry a request that
/// cannot succeed.
///
/// `None` keeps the caller's fallthrough, which is the `500` that integrity
/// faults — damaged, unreadable, or unknown-format key material — must keep.
///
/// Messages either name what the caller asked for or stay generic; detail that
/// belongs to the deployment rather than the request stays in `source`, which
/// the caller attaches and the logs retain.
fn data_plane_kms_error(error: &rustfs_kms::KmsError) -> Option<(S3ErrorCode, String)> {
use rustfs_kms::KmsError as Kms;
let generic = |code: S3ErrorCode| {
let message = ApiError::error_code_to_message(&code);
Some((code, message))
};
match error {
Kms::KeyNotFound { key_id } => Some((
S3ErrorCode::Custom(KMS_KEY_NOT_FOUND_ERROR_CODE.into()),
format!("KMS key not found: {key_id}"),
)),
Kms::KeyVersionNotFound { key_id, version } => Some((
S3ErrorCode::Custom(KMS_KEY_NOT_FOUND_ERROR_CODE.into()),
format!("KMS key version {version} not found for key {key_id}"),
)),
// The key exists but its state forbids the operation (disabled,
// pending deletion). AWS treats an invalid key state as a request
// error, not a server fault.
Kms::InvalidOperation { .. } => Some((S3ErrorCode::InvalidRequest, error.to_string())),
// A policy decision needs a human, so it must be distinguishable from
// the transient classes an SDK retries.
Kms::AccessDenied { .. } => generic(S3ErrorCode::AccessDenied),
// Request-side faults: what was asked for cannot be served as asked.
Kms::ContextMismatch { .. }
| Kms::InvalidKey { .. }
| Kms::ValidationError { .. }
| Kms::UnsupportedAlgorithm { .. }
| Kms::InvalidKeySize { .. } => Some((S3ErrorCode::InvalidRequest, error.to_string())),
// A deployment whose KMS configuration cannot serve the request (for
// example no key named and no default). Actionable, but the detail
// describes the deployment, so it stays out of the response body.
Kms::ConfigurationError { .. } => {
Some((S3ErrorCode::InvalidRequest, "The KMS configuration cannot serve this request".to_string()))
}
// Transient: worth retrying, and must be counted against availability
// rather than against the caller. `IoError` belongs here because it is
// how a backend reports that its key store itself was unreachable —
// rustfs/rustfs#7470 separated that from a missing key precisely so the
// two stop looking alike, and leaving it on the 500 fallthrough would
// erase that distinction again at the S3 boundary.
Kms::BackendError { .. }
| Kms::IoError { .. }
| Kms::OperationTimedOut { .. }
| Kms::OperationCancelled { .. }
| Kms::CredentialsUnavailable { .. }
| Kms::CacheError { .. } => generic(S3ErrorCode::ServiceUnavailable),
// A permanent gap in the configured backend, never a missing resource.
Kms::UnsupportedCapability { .. } => Some((S3ErrorCode::NotImplemented, error.to_string())),
_ => None,
}
}
/// HTTP status of the error codes s3s cannot derive on its own.
///
/// s3s answers `None` for every `Custom` code, which the response layer turns
@@ -693,17 +763,6 @@ impl From<crate::table_catalog::TableCatalogStoreError> for ApiError {
}
}
/// An SSE failure arrives already classified for the S3 data path; carry it over as is.
impl From<SseError> for ApiError {
fn from(err: SseError) -> Self {
ApiError {
code: err.code,
message: err.message,
source: err.source,
}
}
}
impl From<StorageError> for ApiError {
fn from(err: StorageError) -> Self {
record_s3_lock_failure(&err);
+2 -4
View File
@@ -25,7 +25,7 @@ use crate::storage_api::startup::init::{
use crate::storage_api::startup::services::ECStore;
#[cfg(any(feature = "ftps", feature = "webdav", feature = "sftp"))]
use crate::storage_api::startup::services::ServerContextSlot;
use crate::storage_api::startup::sse::init_sse_startup;
use crate::storage_api::startup::sse::log_sse_kms_key_policy_mode;
use crate::{admin, config, startup_runtime_sources, version};
use rustfs_config::{
DEFAULT_BUFFER_MAX_SIZE, DEFAULT_BUFFER_MIN_SIZE, DEFAULT_BUFFER_PROFILE, DEFAULT_BUFFER_UNKNOWN_SIZE, DEFAULT_UPDATE_CHECK,
@@ -539,9 +539,7 @@ pub async fn init_kms_system(config: &config::Config, store: Arc<ECStore>) -> st
// Initialize global KMS service manager (starts in NotConfigured state)
let service_manager = startup_runtime_sources::init_kms_service_manager();
// Settles what SSE reads from the environment. Idempotent, so a second
// embedded server in this process reports nothing new.
init_sse_startup();
log_sse_kms_key_policy_mode();
// A key referenced by any bucket's encryption configuration must never be
// deleted. Register the gate before the service can start so every
-2
View File
@@ -12,9 +12,7 @@ Applies to `rustfs/src/storage/`.
- Never log plaintext customer keys, decrypted key material, or sensitive SSE request payloads.
- Keep SSE behavior aligned with KMS and admin handler integrations.
- SSE key resolution lives in `crates/sse/`; `sse.rs` here only binds it to the running server (`RustfsSseHost`) and translates across the storage-engine boundary. Put new SSE logic in the crate unless it needs ecstore, IAM, or request types.
- Coordinate related changes with:
- `crates/sse/`
- `crates/kms/`
- `rustfs/src/admin/handlers/kms*.rs`
- `crates/ecstore/`
-1
View File
@@ -1954,7 +1954,6 @@ mod tests {
// Drive the real data path: the principal built from this request is
// what carries the audit slot into the SSE code.
let metadata = unreadable_sse_kms_metadata();
use crate::storage::sse::SseKmsPrincipalExt as _;
let envelope = crate::app::RequestEnvelope::from_s3s(&req);
let principal =
crate::storage::sse::SseKmsPrincipal::from_request(&envelope).expect("authenticated request has a principal");
+9243 -607
View File
File diff suppressed because it is too large Load Diff
-233
View File
@@ -252,237 +252,4 @@ mod tests {
println!("✅ Full cycle (generate -> encrypt DEK -> decrypt DEK -> decrypt data) test passed!");
}
#[tokio::test]
async fn test_simple_sse_dek_provider_encrypt_decrypt() {
use std::io::Cursor;
use tokio::io::AsyncReadExt;
// 1. Setup: Create SimpleSseDekProvider with test master key
let provider = TestSseDekProvider::new_with_key([42u8; 32]);
// 2. Generate a data encryption key
let bucket = "test-bucket";
let key = "test-key";
let kms_key_id = "default"; // Key ID is ignored in simple provider
let context = ObjectEncryptionContext::new(bucket.to_string(), key.to_string());
let (data_key, _encrypted_dek) = provider
.generate_sse_dek(&context, kms_key_id)
.await
.expect("Failed to generate DEK");
// 3. Prepare test data (plaintext)
let plaintext = b"Hello, World! This is a test message for encryption and decryption.";
println!("Original plaintext: {:?}", String::from_utf8_lossy(plaintext));
println!("Plaintext length: {} bytes", plaintext.len());
// 4. Encrypt with EncryptReader.
let plaintext_reader = Cursor::new(plaintext.to_vec());
let mut encrypt_reader = EncryptReader::new(plaintext_reader, data_key.plaintext_key, data_key.nonce);
// Read encrypted data
let mut encrypted_data = Vec::new();
encrypt_reader
.read_to_end(&mut encrypted_data)
.await
.expect("Failed to read encrypted data");
println!("Encrypted data length: {} bytes", encrypted_data.len());
println!(
"First 16 bytes of encrypted data: {:02x?}",
&encrypted_data[..16.min(encrypted_data.len())]
);
// Verify encrypted data is different from plaintext
assert_ne!(
&encrypted_data[..plaintext.len()],
plaintext,
"Encrypted data should be different from plaintext"
);
// 5. Decrypt with DecryptReader.
let encrypted_reader = Cursor::new(encrypted_data);
let mut decrypt_reader = DecryptReader::new(encrypted_reader, data_key.plaintext_key, data_key.nonce);
// Read decrypted data
let mut decrypted_data = Vec::new();
decrypt_reader
.read_to_end(&mut decrypted_data)
.await
.expect("Failed to read decrypted data");
println!("Decrypted data: {:?}", String::from_utf8_lossy(&decrypted_data));
println!("Decrypted length: {} bytes", decrypted_data.len());
// 6. Verify decrypted data matches original plaintext
assert_eq!(decrypted_data, plaintext, "Decrypted data should match original plaintext");
println!("✅ Encryption/Decryption test passed!");
}
#[tokio::test]
async fn test_simple_sse_dek_provider_encrypt_decrypt_large_data() {
use std::io::Cursor;
use tokio::io::AsyncReadExt;
// 1. Setup: Create SimpleSseDekProvider with test master key
let provider = TestSseDekProvider::new_with_key([42u8; 32]);
let bucket = "test-bucket";
let key = "test-key-large";
let kms_key_id = "default";
let context = ObjectEncryptionContext::new(bucket.to_string(), key.to_string());
let (data_key, _encrypted_dek) = provider
.generate_sse_dek(&context, kms_key_id)
.await
.expect("Failed to generate DEK");
// Create 1MB of test data
let plaintext_size = 1024 * 1024; // 1MB
let plaintext: Vec<u8> = (0..plaintext_size).map(|i| (i % 256) as u8).collect();
println!("Testing with {} bytes of data", plaintext.len());
// Encrypt.
let plaintext_reader = Cursor::new(plaintext.clone());
let mut encrypt_reader = EncryptReader::new(plaintext_reader, data_key.plaintext_key, data_key.nonce);
let mut encrypted_data = Vec::new();
encrypt_reader
.read_to_end(&mut encrypted_data)
.await
.expect("Failed to encrypt large data");
println!("Encrypted {} bytes to {} bytes", plaintext.len(), encrypted_data.len());
// Decrypt.
let encrypted_reader = Cursor::new(encrypted_data);
let mut decrypt_reader = DecryptReader::new(encrypted_reader, data_key.plaintext_key, data_key.nonce);
let mut decrypted_data = Vec::new();
decrypt_reader
.read_to_end(&mut decrypted_data)
.await
.expect("Failed to decrypt large data");
// Verify
assert_eq!(decrypted_data.len(), plaintext.len(), "Decrypted size should match original");
assert_eq!(decrypted_data, plaintext, "Decrypted data should match original plaintext");
println!("✅ Large data encryption/decryption test passed!");
}
#[tokio::test]
async fn test_simple_sse_dek_provider_different_nonces() {
use std::io::Cursor;
use tokio::io::AsyncReadExt;
// 1. Setup: Create SimpleSseDekProvider with test master key
let provider = TestSseDekProvider::new_with_key([42u8; 32]);
let bucket = "test-bucket";
let key = "test-key";
let kms_key_id = "default";
let context = ObjectEncryptionContext::new(bucket.to_string(), key.to_string());
// Generate two different keys (with different nonces)
let (data_key1, _) = provider
.generate_sse_dek(&context, kms_key_id)
.await
.expect("Failed to generate DEK 1");
let (data_key2, _) = provider
.generate_sse_dek(&context, kms_key_id)
.await
.expect("Failed to generate DEK 2");
// Verify nonces are different
assert_ne!(data_key1.nonce, data_key2.nonce, "Different keys should have different nonces");
// Same plaintext
let plaintext = b"Same plaintext";
// Encrypt with first key.
let reader1 = Cursor::new(plaintext.to_vec());
let mut encrypt_reader1 = EncryptReader::new(reader1, data_key1.plaintext_key, data_key1.nonce);
let mut encrypted1 = Vec::new();
encrypt_reader1.read_to_end(&mut encrypted1).await.unwrap();
// Encrypt with second key.
let reader2 = Cursor::new(plaintext.to_vec());
let mut encrypt_reader2 = EncryptReader::new(reader2, data_key2.plaintext_key, data_key2.nonce);
let mut encrypted2 = Vec::new();
encrypt_reader2.read_to_end(&mut encrypted2).await.unwrap();
// Verify ciphertexts are different (due to different nonces/keys)
assert_ne!(
encrypted1, encrypted2,
"Same plaintext with different nonces should produce different ciphertext"
);
println!("✅ Different nonces produce different ciphertext - test passed!");
}
#[tokio::test]
async fn test_simple_sse_dek_provider_decrypt_with_encrypted_dek() {
use std::io::Cursor;
use tokio::io::AsyncReadExt;
// 1. Setup: Create SimpleSseDekProvider with test master key
let provider = TestSseDekProvider::new_with_key([42u8; 32]);
let bucket = "test-bucket";
let key = "test-key";
let kms_key_id = "default";
let context = ObjectEncryptionContext::new(bucket.to_string(), key.to_string());
// 1. Generate DEK and get encrypted DEK
let (data_key, encrypted_dek) = provider
.generate_sse_dek(&context, kms_key_id)
.await
.expect("Failed to generate DEK");
let original_plaintext_key = data_key.plaintext_key;
let original_nonce = data_key.nonce;
// 2. Simulate storing encrypted_dek and nonce in metadata
// In real scenario, nonce would be stored separately in metadata
// 3. Later, decrypt the DEK
let decrypted_plaintext_key = provider
.decrypt_sse_dek(&encrypted_dek, kms_key_id, &context)
.await
.expect("Failed to decrypt DEK");
// 4. Verify decrypted key matches original
assert_eq!(
decrypted_plaintext_key, original_plaintext_key,
"Decrypted DEK should match original plaintext key"
);
// 5. Use decrypted key to encrypt/decrypt data
let plaintext = b"Test data with decrypted DEK";
// Encrypt with original key.
let reader = Cursor::new(plaintext.to_vec());
let mut encrypt_reader = EncryptReader::new(reader, original_plaintext_key, original_nonce);
let mut encrypted_data = Vec::new();
encrypt_reader.read_to_end(&mut encrypted_data).await.unwrap();
// Decrypt with recovered key (simulating GET operation).
let reader = Cursor::new(encrypted_data);
let mut decrypt_reader = DecryptReader::new(
reader,
decrypted_plaintext_key,
original_nonce, // In real scenario, read from metadata
);
let mut decrypted_data = Vec::new();
decrypt_reader.read_to_end(&mut decrypted_data).await.unwrap();
// Verify
assert_eq!(decrypted_data, plaintext, "Data decrypted with recovered key should match original");
println!("✅ Full cycle (generate -> encrypt DEK -> decrypt DEK -> decrypt data) test passed!");
}
}
+4 -3
View File
@@ -448,8 +448,8 @@ pub(crate) mod s3_api_consumer {
pub(crate) mod sse_consumer {
pub(crate) use super::super::sse::{
EncryptionKeyKind, EncryptionMaterialExt, SseKmsPrincipalExt, bucket_default_write_sse, build_ssec_read_headers,
encryption_material_to_metadata, extract_ssec_params_from_headers, extract_ssekms_context_from_headers, init_sse_startup,
EncryptionKeyKind, bucket_default_write_sse, build_ssec_read_headers, encryption_material_to_metadata,
extract_ssec_params_from_headers, extract_ssekms_context_from_headers, log_sse_kms_key_policy_mode,
map_get_object_reader_error, mark_encrypted_multipart_metadata, multipart_completion_authorization, stored_ssec_metadata,
};
pub(crate) use super::{
@@ -479,7 +479,8 @@ pub(crate) mod tonic_service_consumer {
pub(crate) mod test_consumer {
pub(crate) use super::super::access_owned::object_read::authorize_head_object;
pub(crate) mod sse {
pub(crate) use super::super::super::sse::{RecordingKmsKeyAuthorizer, TestAuthorizationHooks, sse_kms_object_metadata};
pub(crate) use super::super::super::sse::TestAuthorizationHooks;
pub(crate) use super::super::super::sse::tests::{RecordingKmsKeyAuthorizer, sse_kms_object_metadata};
}
pub(crate) use super::super::{
apply_cors_headers, apply_default_lock_retention_metadata, check_preconditions, decode_tags_to_map,
+1 -1
View File
@@ -435,7 +435,7 @@ pub(crate) mod startup {
}
pub(crate) mod sse {
pub(crate) use crate::storage::storage_api::sse_consumer::init_sse_startup;
pub(crate) use crate::storage::storage_api::sse_consumer::log_sse_kms_key_policy_mode;
}
pub(crate) mod notification {
+2 -10
View File
@@ -5497,18 +5497,13 @@ require_source_contains \
# its internal dependency surface is pinned to exactly rustfs-signer so it cannot
# quietly grow storage-side dependencies. Adding any other rustfs-* dependency to
# any of these five crates needs its own adjudication, not a quiet Cargo.toml edit.
# sse sits between the S3 layer and the storage engine and reaches the running
# server only through its `SseHost` port, so its internal dependency surface is
# pinned as well: the KMS it calls, the policy and S3 vocabularies it speaks, and
# the two foundation crates. An edge to ecstore, iam, or the binary would let key
# resolution reach engine or identity state behind the port.
# The pattern matches both TOML dependency spellings: `rustfs-x = ...` and the
# dotted `rustfs-x.workspace = true` form (which previously escaped this guard).
LEAF_CRATE_DEP_HITS_FILE="${TMP_DIR}/leaf_crate_dep_hits.txt"
: >"$LEAF_CRATE_DEP_HITS_FILE"
(
cd "$ROOT_DIR"
for leaf in config credentials crypto io-metrics madmin sse; do
for leaf in config credentials crypto io-metrics madmin; do
manifest="crates/${leaf}/Cargo.toml"
[[ -f "$manifest" ]] || continue
leaf_dep_status=0
@@ -5523,9 +5518,6 @@ LEAF_CRATE_DEP_HITS_FILE="${TMP_DIR}/leaf_crate_dep_hits.txt"
madmin)
rg -v '^[^:]*:[0-9]+:rustfs-signer(\.[a-zA-Z_-]+)* *=' "${TMP_DIR}/leaf_dep_raw.txt" >>"$LEAF_CRATE_DEP_HITS_FILE" || true
;;
sse)
rg -v '^[^:]*:[0-9]+:rustfs-(config|kms|policy|s3-types|utils)(\.[a-zA-Z_-]+)* *=' "${TMP_DIR}/leaf_dep_raw.txt" >>"$LEAF_CRATE_DEP_HITS_FILE" || true
;;
*)
cat "${TMP_DIR}/leaf_dep_raw.txt" >>"$LEAF_CRATE_DEP_HITS_FILE"
;;
@@ -5534,7 +5526,7 @@ LEAF_CRATE_DEP_HITS_FILE="${TMP_DIR}/leaf_crate_dep_hits.txt"
)
if [[ -s "$LEAF_CRATE_DEP_HITS_FILE" ]]; then
report_failure "leaf/pinned-dep crates: config/credentials/crypto take no internal rustfs-* dependency; io-metrics only rustfs-s3-ops (pure contract crate, backlog#1834); madmin only rustfs-signer (admin SDK SigV4 client, #6166); sse only rustfs-config/kms/policy/s3-types/utils (it reaches the server through SseHost); any new edge needs its own adjudication: $(paste -sd '; ' "$LEAF_CRATE_DEP_HITS_FILE")"
report_failure "leaf/pinned-dep crates: config/credentials/crypto take no internal rustfs-* dependency; io-metrics only rustfs-s3-ops (pure contract crate, backlog#1834); madmin only rustfs-signer (admin SDK SigV4 client, #6166); any new edge needs its own adjudication: $(paste -sd '; ' "$LEAF_CRATE_DEP_HITS_FILE")"
fi
# --- ecstore module-level lint blankets (backlog#1823 step 9) ---
-70
View File
@@ -1,70 +0,0 @@
# SSE black-box suite
[中文](README.zh-CN.md)
Business-level tests of server-side encryption against a real `rustfs` process. A case sees only what a client sees: S3 responses, admin API responses, environment variables, and the two startup events the product promises. The suite is written in Python with the standard library only.
## What it answers
- Does each SSE scheme (SSE-S3, SSE-KMS, SSE-C) round-trip bytes and answer with exactly the response headers it should, under every key source the process can start with?
- Are malformed, mismatched and unauthorized requests refused explicitly, with nothing stored?
- Do two server binaries behave the same? Every HTTP exchange inside a case is recorded as an observation vector, and `diff` compares two runs field by field.
## Run
```bash
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py bin add --name head-default --path target/debug/rustfs --commit "$(git rev-parse --short HEAD)" --build default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py diff <run-a> <run-b>
```
`bin add` copies the binary into `target/sse-bb/bin/<name>/` and records its SHA-256, so a later rebuild of `target/debug/rustfs` cannot change what a run tests. A binary built before the `sse_local_master_key` startup event existed is registered with `--no-master-key-event`; the event is then recorded but not asserted.
A run starts one server per process configuration, one after another on the same port, each on a fresh data directory under `target/sse-bb/runs/<run id>/`. The port defaults to 20105; set `SSE_BB_PORT` or pass `--port` to change it. A busy port is an error: the suite does not move to another port and does not stop the process that holds it.
## Process configurations
| Name | Key source |
|---|---|
| `e0` | No KMS, no local master key |
| `e1` | No KMS, local master key loaded |
| `e1x-text`, `e1x-short` | No KMS, local master key set to an invalid value |
| `e2` | Local KMS backend, per-key authorization off |
| `e2e` | Local KMS backend, per-key authorization enforced |
## Output
Each run directory holds `run.json` (binary identity, configurations, counts), `results.jsonl` (one verdict per case and per table row), `observations.jsonl` (the observation vectors), `report.md`, the server logs under `sites/`, and `DONE` with the overall status. A run without `DONE` did not finish and cannot be compared.
A verdict is `pass`, `fail` (an expectation about the product does not hold) or `env-error` (the site is down or a dependency is missing). An environment error is never a pass and never a product failure, and a run with one does not succeed.
Two more verdicts cover behavior the product is known not to meet yet. An expectation written with `known_deviation` stays in the case as the intended behavior: while it does not hold the case reports `known-deviation` and the run still succeeds; once it holds the case reports `deviation-fixed` and the run fails until the marker is turned into a plain `check`.
## Comparing two binaries
`diff` reports three things: verdicts that differ, observations that differ, and cases or exchanges present on one side only. The differences it accepts form a closed list in `diffing.py`; anything else makes the comparison fail. It exits 0 when nothing blocking differs, 1 otherwise, and 2 when the two runs share no exchange, so an empty result cannot be mistaken for agreement.
An observation keeps what is deterministic (status, error code and message, SSE headers, the set of response header names, body hash, the encryption metadata key names stored for the object) and reduces random values (ETags of encrypted objects, version ids, upload ids) to their shape.
Equal behavior of two binaries does not show that one can read what the other wrote. `cross` covers that: the writer binary stores every form the configuration can produce (each scheme, empty to multi-megabyte, single and multipart), the server is stopped, and the reader binary is started on the same data directory and must read every object back.
```bash
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py cross --writer base-default --reader head-default
```
## Checking that the suite can fail
`suite.py selftest` needs no server. It checks the suite's own judgment: that an observation changes when a listed part size, count or order changes and does not change with upload ids or timestamps, that a known deviation tolerates only its one documented difference, and that `diff` reports what it must. Run it after changing `observe.py`, `diffing.py` or `framework.py`.
Two switches exist only to prove that a green run means something:
```bash
# Start the e1 cases on a process without a master key: RT-01, BD-01 and KS-02 must fail.
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default --configs e1 --run-as e1=e0
# Stop the server before the cases: every verdict must be env-error.
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default --configs e1 --sabotage kill-site
```
## Adding a case
Register a function with `@case(id, title, configs)` in one of the `cases_*.py` files. State expectations through the context (`check`, `refused`, `projection`, `roundtrip`, `absent`): each records the evidence that decided it, and a refused write is always followed by a check that nothing was stored. Wrap each input of a table-driven case in `with c.row(name)` so it gets its own verdict. Keep the request sequence deterministic, and put polling inside `observe.quiet()`, because the comparison matches exchanges by their position in the case.
-70
View File
@@ -1,70 +0,0 @@
# SSE 黑盒测试套件
[English](README.md)
针对真实 `rustfs` 进程的服务端加密业务测试。用例只看客户端能看到的东西:S3 响应、admin 接口响应、环境变量,以及产品承诺的两条启动事件。套件用 Python 编写,只依赖标准库。
## 它回答什么问题
- 每种 SSE 方案(SSE-S3、SSE-KMS、SSE-C)在进程可能启动的每种密钥来源下,能否原样读回字节,响应头是否恰好是应有的那几个。
- 格式错误、方案不匹配、无权限的请求是否被明确拒绝,且没有任何东西落盘。
- 两个服务端二进制的行为是否一致。用例里的每次 HTTP 交互都记成一条观测向量,`diff` 逐字段比较两次运行。
## 运行
```bash
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py bin add --name head-default --path target/debug/rustfs --commit "$(git rev-parse --short HEAD)" --build default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py diff <运行甲> <运行乙>
```
`bin add` 把二进制复制到 `target/sse-bb/bin/<名称>/` 并记录 SHA-256,之后重新编译 `target/debug/rustfs` 不会改变一次运行测的是什么。早于 `sse_local_master_key` 启动事件的二进制用 `--no-master-key-event` 登记,这条事件对它只记录、不断言。
一次运行为每种进程配置各启动一个服务端,依次使用同一个端口,每个都用 `target/sse-bb/runs/<运行编号>/` 下全新的数据目录。端口默认 20105,可用 `SSE_BB_PORT` 或 `--port` 修改。端口被占用就报错:套件不会换端口,也不会去停占用它的进程。
## 进程配置
| 名称 | 密钥来源 |
|---|---|
| `e0` | 无 KMS,未设置本地主密钥 |
| `e1` | 无 KMS,本地主密钥已加载 |
| `e1x-text`、`e1x-short` | 无 KMS,本地主密钥取值非法 |
| `e2` | Local KMS 后端,按密钥授权关闭 |
| `e2e` | Local KMS 后端,强制按密钥授权 |
## 输出
每个运行目录包含 `run.json`(二进制身份、配置、计数)、`results.jsonl`(每条用例与每个输入行的结论)、`observations.jsonl`(观测向量)、`report.md`、`sites/` 下的服务端日志,以及写有总体状态的 `DONE`。没有 `DONE` 的运行没有跑完,不能参与比较。
基本结论有三种:`pass`、`fail`(对产品的某条期望不成立)、`env-error`(站点没起来或依赖缺失)。环境错误既不算通过,也不算产品失败;只要出现一条,整次运行就不算成功。
另有两种结论用于产品目前已知达不到的行为。用 `known_deviation` 写下的期望仍然留在用例里,表示应有的行为:它不成立时用例报 `known-deviation`,运行仍算成功;它成立之后用例报 `deviation-fixed`,运行失败,直到把这个标记改成普通的 `check`。
## 比较两个二进制
`diff` 报告三类内容:结论不同、观测不同、只在一边出现的用例或交互。它接受的差异是 `diffing.py` 里的一张封闭清单,清单之外的任何差异都使比较失败。没有阻断性差异时退出码为 0,有则为 1;两次运行没有任何共同交互时为 2,这样"结果为空"不会被当成"一致"。
观测向量保留确定性的内容(状态码、错误码与消息、SSE 响应头、响应头名称集合、正文哈希、对象落盘的加密元数据键名),把随机值(加密对象的 ETag、版本 ID、上传 ID)归约为形态。
两个二进制行为一致,并不能说明一个能读另一个写下的对象。`cross` 负责这一点:写方二进制把该配置能产生的每种落盘形态都写一遍(每种方案、从空对象到数 MiB、单段与多段),停掉服务端,再用读方二进制在同一个数据目录上启动,必须把每个对象原样读回。
```bash
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py cross --writer base-default --reader head-default
```
## 证明套件会失败
`suite.py selftest` 不需要服务端。它检查套件自己的判断力:列举结果里分段的大小、数量或顺序变了,观测必须跟着变,而上传 ID 与时间戳变了不能跟着变;已知偏差只容忍它登记的那一处差异;`diff` 该报的都会报。改动 `observe.py`、`diffing.py` 或 `framework.py` 之后要跑一遍。
有两个开关只用来证明"全绿"是有意义的:
```bash
# 在没有主密钥的进程上跑 e1 的用例:RT-01、BD-01、KS-02 必须失败。
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default --configs e1 --run-as e1=e0
# 在跑用例之前停掉服务端:所有结论必须是 env-error。
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default --configs e1 --sabotage kill-site
```
## 新增用例
在某个 `cases_*.py` 里用 `@case(编号, 标题, 配置)` 注册一个函数。期望通过上下文表达(`check`、`refused`、`projection`、`roundtrip`、`absent`):每个都会记下决定判定的证据,被拒绝的写之后一定跟一次"没有落盘"的检查。表驱动用例的每个输入用 `with c.row(名称)` 包起来,让它有自己的结论。请求序列要保持确定,轮询放进 `observe.quiet()`,因为比较是按交互在用例里的位置对齐的。
@@ -1,126 +0,0 @@
"""CopyObject (CP) and multipart uploads (MP)."""
from __future__ import annotations
import harness as h
from framework import Ctx, case
KEY_A = h.derived_key("ssec a")
KEY_B = h.derived_key("ssec b")
SIZE = 70 * h.KIB
PART = 5 * h.MIB
@case("CP-01", "CopyObject across schemes: the destination scheme alone decides the result", ["e2"])
def cp_01(c: Ctx) -> None:
bucket = c.bucket()
# name: (source write headers, copy-source headers, destination headers, destination read headers, projection)
combos = {
"plain-to-kms": (None, {}, h.sse_kms(h.KEY_2), None, h.projection_kms(h.KEY_2)),
"kms-to-ssec": (h.sse_kms(h.KEY_1), {}, h.ssec(KEY_B), h.ssec(KEY_B), h.projection_c(KEY_B)),
"ssec-to-s3": (h.ssec(KEY_A), h.ssec(KEY_A, h.COPY_SSEC), h.sse_s3(), None, h.projection_s3()),
"ssec-to-ssec": (h.ssec(KEY_A), h.ssec(KEY_A, h.COPY_SSEC), h.ssec(KEY_B), h.ssec(KEY_B), h.projection_c(KEY_B)),
"s3-to-none": (h.sse_s3(), {}, {}, None, {}),
"kms-to-kms": (h.sse_kms(h.KEY_1), {}, h.sse_kms(h.KEY_2), None, h.projection_kms(h.KEY_2)),
}
for name, (source_headers, copy_source, destination, read_headers, want) in combos.items():
with c.row(name):
data = h.plain(f"cp-01 {name}", SIZE)
c.ok(h.put(c.site, bucket, f"{name}-src", data, source_headers), "source PUT")
c.ok(h.copy(c.site, bucket, f"{name}-dst", bucket, f"{name}-src", {**copy_source, **destination}), "CopyObject")
c.read_back(bucket, f"{name}-dst", data, read_headers, want, "destination")
status, head_headers = h.head(c.site, bucket, f"{name}-dst", read_headers)
if c.check(status == 200, f"destination HEAD: HTTP {status}, expected 200"):
c.projection(head_headers, want, "destination HEAD")
c.stored(bucket, f"{name}-dst")
if name == "ssec-to-ssec":
# The data was re-encrypted, not relabelled: the source key no longer opens the destination.
c.refused(h.get(c.site, bucket, f"{name}-dst", h.ssec(KEY_A)), 400, "InvalidRequest",
"destination GET with the source key")
@case("CP-02", "Copying an SSE-C source requires its key", ["e0"])
def cp_02(c: Ctx) -> None:
bucket = c.bucket()
c.ok(h.put(c.site, bucket, "src", h.plain("cp-02", SIZE), h.ssec(KEY_A)), "source PUT")
attempts = {"no-source-key": {}, "wrong-source-key": h.ssec(KEY_B, h.COPY_SSEC)}
for name, copy_source in attempts.items():
with c.row(name):
status, _, body = h.copy(c.site, bucket, f"dst-{name}", bucket, "src", copy_source)
code, message = h.s3_error(body)
c.check(status == 400 and code != "", f"CopyObject: HTTP {status} {code!r} {message[:160]!r}, expected an explicit 400")
c.absent(bucket, f"dst-{name}", "CopyObject")
def _upload(c: Ctx, bucket: str, key: str, parts: list[bytes], create_headers, part_headers, complete_headers):
"""A full multipart upload; returns (create headers, complete headers)."""
status, create_response, upload_id, body = h.create_multipart(c.site, bucket, key, create_headers)
c.need(status == 200 and upload_id, f"CreateMultipartUpload: HTTP {status} {h.s3_error(body)}")
etags = []
for number, data in enumerate(parts, start=1):
part_response, _ = c.ok(h.upload_part(c.site, bucket, key, upload_id, number, data, part_headers), f"UploadPart {number}")
etags.append((number, part_response["etag"]))
complete_response, _ = c.ok(h.complete_multipart(c.site, bucket, key, upload_id, etags, complete_headers),
"CompleteMultipartUpload")
return create_response, complete_response
@case("MP-01", "SSE-KMS multipart upload of three parts", ["e2"])
def mp_01(c: Ctx) -> None:
bucket = c.bucket()
parts = [h.plain("mp-01 part 1", PART), h.plain("mp-01 part 2", PART), h.plain("mp-01 part 3", h.MIB + 17)]
data = b"".join(parts)
want = h.projection_kms(h.KEY_1)
create_response, complete_response = _upload(c, bucket, "object", parts, h.sse_kms(h.KEY_1), None, None)
c.projection(create_response, want, "CreateMultipartUpload")
c.projection(complete_response, want, "CompleteMultipartUpload")
c.read_back(bucket, "object", data, None, want, "completed object")
status, head_headers = h.head(c.site, bucket, "object")
if c.check(status == 200, f"HEAD: HTTP {status}, expected 200"):
c.check(head_headers.get("content-length") == str(len(data)),
f"HEAD: Content-Length {head_headers.get('content-length')}, expected {len(data)}")
etag = head_headers.get("etag", "").strip('"')
c.check(len(etag) == 34 and etag.endswith("-3"), f"HEAD: ETag {etag!r}, expected 32 hex digits followed by -3")
c.stored(bucket, "object")
@case("MP-03", "SSE-C multipart upload", ["e0"])
def mp_03(c: Ctx) -> None:
bucket = c.bucket()
parts = [h.plain("mp-03 part 1", PART), h.plain("mp-03 part 2", h.MIB + 17)]
data = b"".join(parts)
triple, want = h.ssec(KEY_A), h.projection_c(KEY_A)
create_response, complete_response = _upload(c, bucket, "object", parts, triple, triple, triple)
for name, response in (("CreateMultipartUpload", create_response), ("CompleteMultipartUpload", complete_response)):
# Both multipart responses also carry `x-amz-server-side-encryption: AES256`, which belongs to SSE-S3.
c.projection_with_known_extra(response, want, {h.SSE: "AES256"}, name)
c.read_back(bucket, "object", data, triple, want, "completed object")
c.refused(h.get(c.site, bucket, "object"), 400, "InvalidRequest", "GET without the customer key")
c.stored(bucket, "object")
@case("MP-06", "A part whose SSE-C key does not match its upload is refused and does not poison the upload", ["e0"])
def mp_06(c: Ctx) -> None:
bucket = c.bucket()
triple = h.ssec(KEY_A)
part_1, part_2 = h.plain("mp-06 part 1", PART), h.plain("mp-06 part 2", h.KIB)
status, _, upload_id, body = h.create_multipart(c.site, bucket, "ssec", triple)
c.need(status == 200 and upload_id, f"CreateMultipartUpload (SSE-C): HTTP {status} {h.s3_error(body)}")
with c.row("part-without-key"):
c.refused(h.upload_part(c.site, bucket, "ssec", upload_id, 1, part_1), 400, "InvalidRequest", "UploadPart")
with c.row("part-with-another-key"):
c.refused(h.upload_part(c.site, bucket, "ssec", upload_id, 1, part_1, h.ssec(KEY_B)), 400, "InvalidRequest", "UploadPart")
status, _, plain_upload, body = h.create_multipart(c.site, bucket, "plain")
c.need(status == 200 and plain_upload, f"CreateMultipartUpload (plaintext): HTTP {status} {h.s3_error(body)}")
with c.row("key-on-plaintext-upload"):
c.refused(h.upload_part(c.site, bucket, "plain", plain_upload, 1, part_1, triple), 400, "InvalidRequest", "UploadPart")
with c.row("upload-still-usable"):
etags = []
for number, data in ((1, part_1), (2, part_2)):
response, _ = c.ok(h.upload_part(c.site, bucket, "ssec", upload_id, number, data, triple), f"UploadPart {number}")
etags.append((number, response["etag"]))
c.ok(h.complete_multipart(c.site, bucket, "ssec", upload_id, etags, triple), "CompleteMultipartUpload")
c.read_back(bucket, "ssec", part_1 + part_2, triple, h.projection_c(KEY_A), "completed object")
@@ -1,236 +0,0 @@
"""Malformed requests (NG), what SSE-S3 must not reveal (SEC-04), and per-key authorization (AZ)."""
from __future__ import annotations
import base64
import json
import harness as h
import observe
from framework import Ctx, case
KEY_A = h.derived_key("ssec a")
KEY_B = h.derived_key("ssec b")
SIZE = 70 * h.KIB
def _b64(text: str) -> str:
return base64.b64encode(text.encode()).decode()
@case("NG-01", "PutObject refuses malformed combinations of SSE headers", ["e2"])
def ng_01(c: Ctx) -> None:
bucket = c.bucket()
data = h.plain("ng-01", h.KIB)
rows = {
"algorithm-aes128": {h.SSE: "AES128"},
"algorithm-lowercase": {h.SSE: "aes256"},
"ssec-with-aes256": {**h.ssec(KEY_A), h.SSE: "AES256"},
"ssec-with-key-id": {**h.ssec(KEY_A), h.SSE_KEY_ID: h.KEY_1},
"key-id-without-algorithm": {h.SSE_KEY_ID: h.KEY_1},
"aes256-with-key-id": {h.SSE: "AES256", h.SSE_KEY_ID: h.KEY_1},
"context-without-algorithm": {h.SSE_CONTEXT: _b64('{"dept":"a"}')},
"context-not-base64": {h.SSE: "aws:kms", h.SSE_CONTEXT: "!!! not base64 !!!"},
"context-json-array": {h.SSE: "aws:kms", h.SSE_CONTEXT: _b64('["dept","a"]')},
"context-key-equals-bucket": {h.SSE: "aws:kms", h.SSE_CONTEXT: _b64(json.dumps({bucket: "elsewhere"}))},
}
for name, headers in rows.items():
with c.row(name):
c.refused(h.put(c.site, bucket, name, data, headers), 400, "InvalidArgument", "PUT")
c.absent(bucket, name, "PUT")
@case("NG-02", "PutObject refuses an incomplete or inconsistent SSE-C triple", ["e0"])
def ng_02(c: Ctx) -> None:
bucket = c.bucket()
data = h.plain("ng-02", h.KIB)
triple = h.ssec(KEY_A)
algorithm, key, md5 = f"{h.SSEC}algorithm", f"{h.SSEC}key", f"{h.SSEC}key-md5"
short = h.derived_key("short", 16)
rows = {
"algorithm-only": {algorithm: "AES256"},
"key-only": {key: triple[key]},
"md5-missing": {algorithm: "AES256", key: triple[key]},
"key-not-base64": {**triple, key: "!!! not base64 !!!"},
"key-16-bytes": h.ssec(short),
"md5-of-another-key": {**triple, md5: h.key_md5(KEY_B)},
"algorithm-aes128": {**triple, algorithm: "AES128"},
}
for name, headers in rows.items():
with c.row(name):
# A partial triple that is treated as "no SSE-C" would store the object in plaintext.
c.refused(h.put(c.site, bucket, name, data, headers), 400, "InvalidRequest", "PUT")
c.absent(bucket, name, "PUT")
@case("NG-03", "GetObject refuses read headers that do not match how the object was stored", ["e2"])
def ng_03(c: Ctx) -> None:
bucket = c.bucket()
objects = {"ssec": h.ssec(KEY_A), "kms": h.sse_kms(h.KEY_1), "plain": None}
data = {name: h.plain(f"ng-03 {name}", SIZE) for name in objects}
for name, headers in objects.items():
c.ok(h.put(c.site, bucket, name, data[name], headers), f"{name} PUT")
md5_only_matches = {**h.ssec(KEY_B), f"{h.SSEC}key-md5": h.key_md5(KEY_A)}
rows = {
"ssec-without-key": ("ssec", None, "InvalidRequest"),
"ssec-with-another-key": ("ssec", h.ssec(KEY_B), "InvalidRequest"),
"ssec-right-md5-wrong-key": ("ssec", md5_only_matches, "InvalidRequest"),
"kms-with-ssec": ("kms", h.ssec(KEY_A), "InvalidArgument"),
"plain-with-ssec": ("plain", h.ssec(KEY_A), "InvalidRequest"),
"plain-with-aes256": ("plain", h.sse_s3(), "InvalidArgument"),
# No customer key came with the request, and that is what the refusal reports; the aws:kms header is not reached.
"ssec-with-aws-kms": ("ssec", h.sse_kms(), "InvalidRequest"),
}
for name, (target, headers, code) in rows.items():
with c.row(name):
response = h.get(c.site, bucket, target, headers)
c.refused(response, 400, code, "GET")
c.check(data[target][:64] not in response[2], "the refused GET returned object bytes")
@case("NG-04", "SSE-KMS with a key that does not exist", ["e2"])
def ng_04(c: Ctx) -> None:
bucket = c.bucket()
c.refused(h.put(c.site, bucket, "object", h.plain("ng-04", h.KIB), h.sse_kms(h.MISSING_KEY)), 400,
"KMS.NotFoundException", "PUT")
c.absent(bucket, "object", "PUT")
@case("SEC-04", "No SSE-S3 response names the key that wraps the data key", ["e2"])
def sec_04(c: Ctx) -> None:
bucket = c.bucket()
data = h.plain("sec-04", SIZE)
part = h.plain("sec-04 part", 5 * h.MIB)
responses = {}
responses["PutObject"] = h.put(c.site, bucket, "object", data, h.sse_s3())
responses["GetObject"] = h.get(c.site, bucket, "object")
status, head_headers = h.head(c.site, bucket, "object")
responses["HeadObject"] = (status, head_headers, b"")
responses["CopyObject"] = h.copy(c.site, bucket, "copy", bucket, "object", h.sse_s3())
status, create_headers, upload_id, body = h.create_multipart(c.site, bucket, "multipart", h.sse_s3())
responses["CreateMultipartUpload"] = (status, create_headers, body)
c.need(status == 200 and upload_id, f"CreateMultipartUpload: HTTP {status} {h.s3_error(body)}")
responses["UploadPart"] = h.upload_part(c.site, bucket, "multipart", upload_id, 1, part)
responses["ListParts"] = h.list_parts(c.site, bucket, "multipart", upload_id)
etag = responses["UploadPart"][1].get("etag", "")
responses["CompleteMultipartUpload"] = h.complete_multipart(c.site, bucket, "multipart", upload_id, [(1, etag)])
for name, (status, headers, body) in responses.items():
c.check(status == 200, f"{name}: HTTP {status}, expected 200")
c.check(h.SSE_KEY_ID not in headers, f"{name}: the response carries {h.SSE_KEY_ID}: {headers.get(h.SSE_KEY_ID)!r}")
if name != "GetObject":
c.check(h.DEFAULT_KEY.encode() not in body, f"{name}: the response body names the default key {h.DEFAULT_KEY}")
for header, value in headers.items():
c.check(h.DEFAULT_KEY not in value, f"{name}: header {header} names the default key {h.DEFAULT_KEY}")
# ---------------------------------------------------------------------------
# Per-key authorization: three identities with full S3 access and different KMS grants
# ---------------------------------------------------------------------------
USER_SECRET = "sse-blackbox-user-secret"
KMS_ARN_FORMS = ("arn:rustfs:kms:::key/", "arn:aws:kms:::key/")
KMS_ALL_KEYS = "arn:aws:kms:::*"
_identities: dict[str, dict] = {}
def _policy(arn: str, name: str) -> dict:
s3 = {"Effect": "Allow", "Action": ["s3:*"], "Resource": ["arn:aws:s3:::*"]}
use = ["kms:GenerateDataKey", "kms:Decrypt"]
extra = {
# U1: may use KEY_1 only.
"u1": [{"Effect": "Allow", "Action": use, "Resource": [f"{arn}{h.KEY_1}"]}],
# U2: full S3 access and no KMS permission at all.
"u2": [],
# U3: may use every key, with an explicit deny on KEY_1.
"u3": [{"Effect": "Allow", "Action": ["kms:*"], "Resource": [KMS_ALL_KEYS]},
{"Effect": "Deny", "Action": use, "Resource": [f"{arn}{h.KEY_1}"]}],
}[name]
return {"Version": "2012-10-17", "Statement": [s3, *extra]}
def _users(c: Ctx) -> dict[str, tuple[str, str]]:
"""Create the three identities once per site. Provisioning is setup, so it stays out of the observations."""
cache_key = str(c.site.root)
if cache_key in _identities:
return _identities[cache_key]
users = {}
with observe.quiet():
for name in ("u1", "u2", "u3"):
user, policy_name = f"bb-{name}", f"bb-policy-{name}"
last = None
for arn in KMS_ARN_FORMS:
last = h.iam_add_policy(c.site, policy_name, _policy(arn, name))
if last[0] == 200:
break
if last[0] != 200:
raise h.EnvError(f"add policy {policy_name}: HTTP {last[0]} {str(last[1])[:200]}")
for step, (status, body) in (("add user", h.iam_add_user(c.site, user, USER_SECRET)),
("attach policy", h.iam_attach_policy(c.site, user, policy_name))):
if status != 200:
raise h.EnvError(f"{step} {user}: HTTP {status} {str(body)[:200]}")
users[name] = (user, USER_SECRET)
_identities[cache_key] = users
return users
@case("AZ-01", "Under enforcement a caller needs a KMS grant on the key the request resolves to", ["e2e"])
def az_01(c: Ctx) -> None:
bucket = c.bucket()
users = _users(c)
stored = {key: h.plain(f"az-01 admin {key}", SIZE) for key in (h.KEY_1, h.KEY_2)}
for key, data in stored.items():
c.ok(h.put(c.site, bucket, f"admin-{key}", data, h.sse_kms(key)), f"admin PUT under {key}")
# identity -> key -> allowed
matrix = {
"u1": {h.KEY_1: True, h.KEY_2: False},
"u2": {h.KEY_1: False, h.KEY_2: False},
"u3": {h.KEY_1: False, h.KEY_2: True},
}
for name, grants in matrix.items():
for key, allowed in grants.items():
with c.row(f"{name}-{key}"):
data = h.plain(f"az-01 {name} {key}", SIZE)
written = h.put(c.site, bucket, f"{name}-{key}", data, h.sse_kms(key), creds=users[name])
read = h.get(c.site, bucket, f"admin-{key}", creds=users[name])
if allowed:
put_headers, _ = c.ok(written, "PUT")
c.projection(put_headers, h.projection_kms(key), "PUT")
get_headers, body = c.ok(read, "GET")
c.bytes_equal(get_headers, body, stored[key], "GET")
else:
c.refused(written, 403, "AccessDenied", "PUT")
c.absent(bucket, f"{name}-{key}", "PUT")
c.refused(read, 403, "AccessDenied", "GET")
c.check(stored[key][:64] not in read[2], "the refused GET returned object bytes")
@case("AZ-02", "Enforcement leaves schemes that name no KMS key alone", ["e2e"])
def az_02(c: Ctx) -> None:
bucket = c.bucket()
creds = _users(c)["u2"]
for name, write_headers, read_headers, want in (("sse-s3", h.sse_s3(), None, h.projection_s3()),
("sse-c", h.ssec(KEY_A), h.ssec(KEY_A), h.projection_c(KEY_A))):
with c.row(name):
data = h.plain(f"az-02 {name}", SIZE)
put_headers, _ = c.ok(h.put(c.site, bucket, name, data, write_headers, creds=creds), "PUT")
c.projection(put_headers, want, "PUT")
get_headers, body = c.ok(h.get(c.site, bucket, name, read_headers, creds=creds), "GET")
c.projection(get_headers, want, "GET")
c.bytes_equal(get_headers, body, data, "GET")
@case("AZ-03", "A denied caller cannot tell an existing key from a missing one", ["e2e"])
def az_03(c: Ctx) -> None:
bucket = c.bucket()
creds = _users(c)["u2"]
data = h.plain("az-03", h.KIB)
existing = h.put(c.site, bucket, "denied-existing", data, h.sse_kms(h.KEY_1), creds=creds)
missing = h.put(c.site, bucket, "denied-missing", data, h.sse_kms(h.MISSING_KEY), creds=creds)
c.refused(existing, 403, "AccessDenied", "denied PUT under an existing key")
c.refused(missing, 403, "AccessDenied", "denied PUT under a missing key")
c.check(h.s3_error(existing[2]) == h.s3_error(missing[2]),
f"the two denials differ: {h.s3_error(existing[2])} != {h.s3_error(missing[2])}")
c.absent(bucket, "denied-missing", "denied PUT under a missing key")
# The administrator, who is authorized, does learn that the key is missing.
c.refused(h.put(c.site, bucket, "admin-missing", data, h.sse_kms(h.MISSING_KEY)), 400, "KMS.NotFoundException",
"authorized PUT under a missing key")
@@ -1,139 +0,0 @@
"""Round trips and response projections (RT) and bucket default encryption (BD)."""
from __future__ import annotations
import json
import harness as h
from framework import Ctx, case
KEY_A = h.derived_key("ssec a")
SIZE = 70 * h.KIB
@case("RT-01", "SSE-S3 under the local master key", ["e1"])
def rt_01(c: Ctx) -> None:
bucket = c.bucket()
c.roundtrip(bucket, "object", h.plain("rt-01", SIZE), h.sse_s3(), None, h.projection_s3(), "SSE-S3")
@case("RT-02", "SSE-S3 under the KMS default key", ["e2"])
def rt_02(c: Ctx) -> None:
bucket = c.bucket()
c.roundtrip(bucket, "object", h.plain("rt-02", SIZE), h.sse_s3(), None, h.projection_s3(), "SSE-S3")
c.check(h.DEFAULT_KEY in h.key_list(c.site), f"the KMS key list no longer contains the default key {h.DEFAULT_KEY}")
@case("RT-03", "SSE-KMS without a key id resolves to the KMS default key", ["e2"])
def rt_03(c: Ctx) -> None:
bucket = c.bucket()
status, body = h.kms_status(c.site)
c.check(status == 200 and h.DEFAULT_KEY in json.dumps(body),
f"KMS status does not report the default key {h.DEFAULT_KEY}: HTTP {status} {str(body)[:200]}")
c.roundtrip(bucket, "object", h.plain("rt-03", SIZE), h.sse_kms(), None, h.projection_kms(h.DEFAULT_KEY), "SSE-KMS")
@case("RT-04", "SSE-KMS with an explicit key that is not the default", ["e2"])
def rt_04(c: Ctx) -> None:
bucket = c.bucket()
# KEY_1 differs from the default key, so a response naming the wrong key cannot pass.
c.roundtrip(bucket, "object", h.plain("rt-04", SIZE), h.sse_kms(h.KEY_1), None, h.projection_kms(h.KEY_1), "SSE-KMS")
@case("RT-06", "SSE-C needs no server-side key source", ["e0"])
def rt_06(c: Ctx) -> None:
bucket = c.bucket()
c.roundtrip(bucket, "object", h.plain("rt-06", SIZE), h.ssec(KEY_A), h.ssec(KEY_A), h.projection_c(KEY_A), "SSE-C")
def _schemes(c: Ctx) -> list[tuple[str, dict, dict | None, dict]]:
"""(name, write headers, read headers, projection) for the schemes this configuration can serve."""
schemes = [("sse-s3", h.sse_s3(), None, h.projection_s3()), ("sse-c", h.ssec(KEY_A), h.ssec(KEY_A), h.projection_c(KEY_A))]
if c.config == "e2":
schemes = [("sse-kms", h.sse_kms(h.KEY_1), None, h.projection_kms(h.KEY_1))]
return schemes
@case("RT-07", "Object sizes 0, 1 and 70 KiB under each scheme", ["e1", "e2"])
def rt_07(c: Ctx) -> None:
bucket = c.bucket()
expected = {}
for name, write_headers, read_headers, want in _schemes(c):
for size in (0, 1, SIZE):
key = f"{name}-{size}"
with c.row(key):
data = h.plain(f"rt-07 {key}", size)
c.roundtrip(bucket, key, data, write_headers, read_headers, want, key)
expected[key] = size
listed = h.list_objects(c.site, bucket)
for key, size in expected.items():
c.check(listed.get(key) == size, f"ListObjectsV2 reports size {listed.get(key)} for {key}, expected the plaintext size {size}")
@case("RT-08", "Range reads of an SSE-KMS object", ["e2"])
def rt_08(c: Ctx) -> None:
bucket = c.bucket()
data = h.plain("rt-08", SIZE)
c.ok(h.put(c.site, bucket, "object", data, h.sse_kms(h.KEY_1)), "PUT")
last = SIZE - 1
ranges = {
"first-byte": ("bytes=0-0", 0, 0),
"last-byte": (f"bytes={last}-{last}", last, last),
"suffix-100": ("bytes=-100", SIZE - 100, last),
"across-8k": ("bytes=8000-8500", 8000, 8500),
}
for name, (spec, first, end) in ranges.items():
with c.row(name):
status, headers, body = h.get(c.site, bucket, "object", range_=spec)
c.check(status == 206, f"Range {spec}: HTTP {status}, expected 206")
c.check(headers.get("content-range") == f"bytes {first}-{end}/{SIZE}",
f"Range {spec}: Content-Range {headers.get('content-range')!r}, expected 'bytes {first}-{end}/{SIZE}'")
c.check(body == data[first:end + 1],
f"Range {spec}: body sha256 {h.sha256_hex(body)[:16]} len {len(body)} is not the plaintext slice")
c.projection(headers, h.projection_kms(h.KEY_1), f"Range {spec}")
@case("BD-01", "Bucket default AES256 encrypts a request that names no scheme", ["e1"])
def bd_01(c: Ctx) -> None:
bucket = c.bucket()
c.ok(h.put_bucket_encryption(c.site, bucket, "AES256"), "PutBucketEncryption")
c.roundtrip(bucket, "object", h.plain("bd-01", SIZE), None, None, h.projection_s3(), "no-header PUT")
@case("BD-02", "Bucket default aws:kms without a key id pins the KMS default key", ["e2"])
def bd_02(c: Ctx) -> None:
bucket = c.bucket()
c.ok(h.put_bucket_encryption(c.site, bucket, "aws:kms"), "PutBucketEncryption")
status, config, _ = h.get_bucket_encryption(c.site, bucket)
c.check(status == 200 and config.get("algorithm") == "aws:kms" and config.get("kms_key_id") == h.DEFAULT_KEY,
f"GetBucketEncryption: HTTP {status} {config}, expected aws:kms with KMSMasterKeyID {h.DEFAULT_KEY}")
c.roundtrip(bucket, "object", h.plain("bd-02", SIZE), None, None, h.projection_kms(h.DEFAULT_KEY), "no-header PUT")
@case("BD-03", "Explicit AES256 wins over a bucket default of aws:kms", ["e2"])
def bd_03(c: Ctx) -> None:
bucket = c.bucket()
c.ok(h.put_bucket_encryption(c.site, bucket, "aws:kms", h.KEY_1), "PutBucketEncryption")
# The projection has no key id: the request did not inherit the bucket's KMS key.
c.roundtrip(bucket, "object", h.plain("bd-03", SIZE), h.sse_s3(), None, h.projection_s3(), "explicit AES256")
@case("BD-04", "SSE-C is accepted under a bucket default and stays SSE-C", ["e1", "e2"])
def bd_04(c: Ctx) -> None:
bucket = c.bucket()
algorithm = "AES256" if c.config == "e1" else "aws:kms"
c.ok(h.put_bucket_encryption(c.site, bucket, algorithm), "PutBucketEncryption")
data = h.plain("bd-04", SIZE)
c.roundtrip(bucket, "object", data, h.ssec(KEY_A), h.ssec(KEY_A), h.projection_c(KEY_A), "SSE-C")
c.refused(h.get(c.site, bucket, "object"), 400, "InvalidRequest", "GET without the customer key")
@case("BD-06", "Removing the bucket default stops encrypting new objects and keeps old ones readable", ["e1"])
def bd_06(c: Ctx) -> None:
bucket = c.bucket()
before, after = h.plain("bd-06 before", SIZE), h.plain("bd-06 after", SIZE)
c.ok(h.put_bucket_encryption(c.site, bucket, "AES256"), "PutBucketEncryption")
c.roundtrip(bucket, "before", before, None, None, h.projection_s3(), "object written under the default")
c.ok(h.delete_bucket_encryption(c.site, bucket), "DeleteBucketEncryption", want=204)
c.roundtrip(bucket, "after", after, None, None, {}, "object written after the default was removed")
c.read_back(bucket, "before", before, None, h.projection_s3(), "object written under the default, read afterwards")
-114
View File
@@ -1,114 +0,0 @@
"""Key sources and run-time state (KS): a write follows the state now, a read follows what the object recorded."""
from __future__ import annotations
import harness as h
from framework import CONFIGS, Ctx, case
KEY_A = h.derived_key("ssec a")
SIZE = 70 * h.KIB
MASTER_KEY_EVENT = "sse_local_master_key"
POLICY_MODE_EVENT = "sse_kms_key_policy_mode"
NO_KMS_MESSAGE = "SSE-KMS requires a configured and running KMS service"
def _master_key_state(c: Ctx, want: str, level: str | None = None) -> None:
"""Exactly one startup event states how the local master key was found."""
records = c.startup_event(MASTER_KEY_EVENT)
if not c.master_key_event:
return # recorded for the comparison; this binary predates the event
if c.check(len(records) == 1, f"{len(records)} {MASTER_KEY_EVENT} events in the startup log, expected exactly 1"):
c.check(records[0].get("state") == want, f"{MASTER_KEY_EVENT} state {records[0].get('state')!r}, expected {want!r}")
if level:
got = str(records[0].get("level", "")).lower()
c.check(got == level, f"{MASTER_KEY_EVENT} level {got!r}, expected {level!r}")
def _ssec_roundtrip(c: Ctx, bucket: str) -> None:
c.roundtrip(bucket, "ssec", h.plain(f"{c.case_id} ssec", SIZE), h.ssec(KEY_A), h.ssec(KEY_A), h.projection_c(KEY_A), "SSE-C")
@case("KS-01", "No KMS and no local master key: managed writes are refused, SSE-C works", ["e0"])
def ks_01(c: Ctx) -> None:
bucket = c.bucket()
_master_key_state(c, "not_set")
data = h.plain("ks-01", SIZE)
c.refused(h.put(c.site, bucket, "sse-s3", data, h.sse_s3()), 400, "InvalidRequest", "SSE-S3 PUT",
message_has="RUSTFS_SSE_S3_MASTER_KEY")
c.absent(bucket, "sse-s3", "SSE-S3 PUT")
c.refused(h.put(c.site, bucket, "sse-kms", data, h.sse_kms()), 400, "InvalidRequest", "SSE-KMS PUT",
message_has=NO_KMS_MESSAGE)
c.absent(bucket, "sse-kms", "SSE-KMS PUT")
_ssec_roundtrip(c, bucket)
@case("KS-02", "Local master key loaded: SSE-S3 works, SSE-KMS is refused and never downgraded", ["e1"])
def ks_02(c: Ctx) -> None:
bucket = c.bucket()
_master_key_state(c, "loaded")
c.roundtrip(bucket, "sse-s3", h.plain("ks-02 s3", SIZE), h.sse_s3(), None, h.projection_s3(), "SSE-S3")
# An object that says aws:kms while only the local master key protects it would be a silent downgrade.
c.refused(h.put(c.site, bucket, "sse-kms", h.plain("ks-02 kms", SIZE), h.sse_kms()), 400, "InvalidRequest", "SSE-KMS PUT")
c.absent(bucket, "sse-kms", "SSE-KMS PUT")
@case("KS-03", "An invalid local master key neither stops the process nor leaks its value", ["e1x-text", "e1x-short"])
def ks_03(c: Ctx) -> None:
bucket = c.bucket()
_master_key_state(c, "invalid", level="error")
configured = CONFIGS[c.config](c.site)["RUSTFS_SSE_S3_MASTER_KEY"]
fragments = {configured, configured[:12], configured[-12:]}
response = h.put(c.site, bucket, "sse-s3", h.plain("ks-03", SIZE), h.sse_s3())
c.refused(response, 400, "InvalidRequest", "SSE-S3 PUT")
c.absent(bucket, "sse-s3", "SSE-S3 PUT")
for fragment in fragments:
c.check(fragment.encode() not in response[2], "the SSE-S3 refusal repeats part of the configured master key value")
c.check(fragment not in h.log_text(c.site), "the server log repeats part of the configured master key value")
_ssec_roundtrip(c, bucket)
@case("KS-04", "The startup log states whether per-key authorization is enforced", ["e2", "e2e"])
def ks_04(c: Ctx) -> None:
records = c.startup_event(POLICY_MODE_EVENT)
want_enforced, want_level = ("true", "info") if c.config == "e2e" else ("false", "warn")
if c.check(len(records) == 1, f"{len(records)} {POLICY_MODE_EVENT} events in the startup log, expected exactly 1"):
enforced = str(records[0].get("enforced")).lower()
level = str(records[0].get("level", "")).lower()
c.check(enforced == want_enforced, f"{POLICY_MODE_EVENT} enforced={enforced}, expected {want_enforced}")
c.check(level == want_level, f"{POLICY_MODE_EVENT} level {level!r}, expected {want_level!r}")
@case("KS-06", "Stopping and starting KMS at run time", ["e2"], last=True)
def ks_06(c: Ctx) -> None:
bucket = c.bucket()
kms_data, ssec_data, plain_data = h.plain("ks-06 kms", SIZE), h.plain("ks-06 ssec", SIZE), h.plain("ks-06 plain", SIZE)
want = h.projection_kms(h.KEY_1)
c.ok(h.put(c.site, bucket, "kms", kms_data, h.sse_kms(h.KEY_1)), "SSE-KMS PUT")
c.ok(h.put(c.site, bucket, "ssec", ssec_data, h.ssec(KEY_A)), "SSE-C PUT")
c.ok(h.put(c.site, bucket, "plain", plain_data), "plaintext PUT")
status, body = h.kms_stop(c.site)
c.need(status == 200, f"KMS stop: HTTP {status} {str(body)[:200]}")
try:
with c.row("while-stopped"):
status, _, body = h.put(c.site, bucket, "kms-while-stopped", kms_data, h.sse_kms(h.KEY_1))
c.check(status == 503 and h.s3_error(body)[0] != "",
f"SSE-KMS PUT while KMS is stopped: HTTP {status} {h.s3_error(body)}, expected an explicit 503")
c.absent(bucket, "kms-while-stopped", "SSE-KMS PUT while KMS is stopped")
status, _, body = h.get(c.site, bucket, "kms")
c.check(status == 503 and h.s3_error(body)[0] != "",
f"GET of the SSE-KMS object while KMS is stopped: HTTP {status} {h.s3_error(body)}, expected an explicit 503")
c.check(kms_data[:64] not in body, "the refused GET returned object bytes")
status, head_headers = h.head(c.site, bucket, "kms")
if c.check(status == 200, f"HEAD of the SSE-KMS object while KMS is stopped: HTTP {status}, expected 200"):
c.projection(head_headers, want, "HEAD while KMS is stopped")
c.read_back(bucket, "ssec", ssec_data, h.ssec(KEY_A), h.projection_c(KEY_A), "SSE-C object while KMS is stopped")
c.read_back(bucket, "plain", plain_data, None, {}, "plaintext object while KMS is stopped")
finally:
status, body = h.kms_start(c.site)
c.need(status == 200, f"KMS start: HTTP {status} {str(body)[:200]}")
h.wait_kms_healthy(c.site)
with c.row("after-restart"):
# The process was not restarted: the running server must pick the service up again.
c.read_back(bucket, "kms", kms_data, None, want, "SSE-KMS object after KMS came back")
c.roundtrip(bucket, "kms-after", h.plain("ks-06 after", SIZE), h.sse_kms(h.KEY_1), None, want, "new SSE-KMS object")
-114
View File
@@ -1,114 +0,0 @@
"""Compare two finished runs: verdicts, observation vectors, and which cases exist on one side only."""
from __future__ import annotations
import json
from pathlib import Path
# Differences that a comparison between two builds is allowed to show. The list is closed: anything else fails it.
# - the `sse_local_master_key` startup event exists only from the build that introduced it. The difference is
# accepted only when one binary is registered as predating the event and that side recorded none; two binaries
# that both have the event must agree on it.
ALLOWED_NOTES = {("startup-event", "sse_local_master_key")}
def _predates_event(run: dict) -> bool:
return not run["meta"]["binary"].get("master_key_event", True)
def _load_jsonl(path: Path) -> list[dict]:
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
def load_run(run_dir: Path) -> dict:
done = run_dir / "DONE"
if not done.exists():
raise SystemExit(f"{run_dir}: no DONE marker; the run did not finish and cannot be compared")
return {
"dir": run_dir,
"meta": json.loads((run_dir / "run.json").read_text()),
"results": _load_jsonl(run_dir / "results.jsonl"),
"observations": _load_jsonl(run_dir / "observations.jsonl"),
}
def _result_key(r: dict) -> tuple:
return (r["config"], r["case"], r.get("row", ""))
def _observation_key(o: dict) -> tuple:
return (o["config"], o["case"], o["kind"], o["seq"], o.get("name", ""))
def _field_differences(a: dict, b: dict) -> list[str]:
out = []
for name in sorted(set(a) | set(b)):
if a.get(name) != b.get(name):
out.append(f"{name}: {json.dumps(a.get(name), sort_keys=True)} != {json.dumps(b.get(name), sort_keys=True)}")
return out
def compare(run_a: dict, run_b: dict) -> dict:
"""Returns {'blocking': [...], 'allowed': [...], 'compared': {...}}; an empty 'blocking' list means equal."""
blocking: list[str] = []
allowed: list[str] = []
results_a = {_result_key(r): r for r in run_a["results"]}
results_b = {_result_key(r): r for r in run_b["results"]}
for key in sorted(set(results_a) | set(results_b)):
label = "/".join(str(part) for part in key if part != "")
if key not in results_a or key not in results_b:
# A case that disappears would otherwise read as "fewer failures".
blocking.append(f"result only in {'B' if key not in results_a else 'A'}: {label}")
elif results_a[key]["verdict"] != results_b[key]["verdict"]:
blocking.append(f"verdict differs: {label}: {results_a[key]['verdict']} != {results_b[key]['verdict']}")
obs_a = {_observation_key(o): o for o in run_a["observations"]}
obs_b = {_observation_key(o): o for o in run_b["observations"]}
for key in sorted(set(obs_a) | set(obs_b), key=str):
config, case, kind, seq, name = key
label = f"{config}/{case} {kind} #{seq} {name}".rstrip()
is_allowed = (kind, name) in ALLOWED_NOTES and any(
_predates_event(run) and observations.get(key, {}).get("value") == []
for run, observations in ((run_a, obs_a), (run_b, obs_b)))
if key not in obs_a or key not in obs_b:
side = "B" if key not in obs_a else "A"
present = obs_b[key] if key not in obs_a else obs_a[key]
what = present.get("request") or json.dumps(present.get("value"), sort_keys=True)
(allowed if is_allowed else blocking).append(f"observation only in {side}: {label}: {what}")
continue
differences = _field_differences(obs_a[key], obs_b[key])
if differences:
what = obs_a[key].get("request") or obs_a[key].get("name", "")
entry = f"observation differs: {label} [{obs_a[key].get('op', kind)} {what}]\n " + "\n ".join(differences)
(allowed if is_allowed else blocking).append(entry)
return {
"blocking": blocking,
"allowed": allowed,
"compared": {
"results": len(set(results_a) & set(results_b)),
"observations": len(set(obs_a) & set(obs_b)),
"exchanges": sum(1 for k in set(obs_a) & set(obs_b) if k[2] == "exchange"),
},
}
def render(run_a: dict, run_b: dict, outcome: dict) -> str:
def identity(run: dict) -> str:
binary = run["meta"]["binary"]
return f"{binary.get('name')} (commit {binary.get('commit')}, build {binary.get('build')}, sha256 {binary.get('sha256', '')[:12]})"
compared = outcome["compared"]
lines = [
f"A: {run_a['dir'].name} {identity(run_a)}",
f"B: {run_b['dir'].name} {identity(run_b)}",
f"compared: {compared['results']} verdicts, {compared['observations']} observations ({compared['exchanges']} HTTP exchanges)",
f"blocking differences: {len(outcome['blocking'])}",
f"allowed differences: {len(outcome['allowed'])}",
]
for title, entries in (("BLOCKING", outcome["blocking"]), ("ALLOWED", outcome["allowed"])):
if entries:
lines.append(f"\n== {title} ==")
lines.extend(f"- {entry}" for entry in entries)
return "\n".join(lines) + "\n"
-223
View File
@@ -1,223 +0,0 @@
"""Case registry, process configurations, and the context a case uses to state what it expects.
A case never asserts with a bare `assert`: `Ctx.check` collects every expectation that does not hold, so one run
reports all of them, and the message always names the evidence that decided it.
"""
from __future__ import annotations
import base64
import contextlib
from dataclasses import dataclass, field
import harness as h
import observe
PASS, FAIL, ENV_ERROR, SKIP = "pass", "fail", "env-error", "skip"
# A case whose only unmet expectations are known deviations, and one whose known deviation no longer reproduces.
KNOWN, FIXED = "known-deviation", "deviation-fixed"
# ---------------------------------------------------------------------------
# Process configurations: where the key that wraps data keys comes from
# ---------------------------------------------------------------------------
def _local_kms(site: h.Site) -> dict[str, str]:
return {
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": str(site.key_dir),
"RUSTFS_KMS_DEFAULT_KEY_ID": h.DEFAULT_KEY,
"RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS": "true",
"RUSTFS_KMS_PROBE_INTERVAL_SECS": str(h.KMS_PROBE_INTERVAL),
}
def master_key() -> str:
return base64.b64encode(h.derived_key("local master key")).decode()
CONFIGS = {
# No KMS and no local master key.
"e0": lambda site: {},
# No KMS, local master key loaded.
"e1": lambda site: {"RUSTFS_SSE_S3_MASTER_KEY": master_key()},
# No KMS, local master key set to a value that is not base64.
"e1x-text": lambda site: {"RUSTFS_SSE_S3_MASTER_KEY": "zz#sse-blackbox-invalid-value#zz"},
# No KMS, local master key that decodes to 16 bytes instead of 32.
"e1x-short": lambda site: {"RUSTFS_SSE_S3_MASTER_KEY": base64.b64encode(h.derived_key("short", 16)).decode()},
# Local KMS backend, per-key authorization off (the default).
"e2": _local_kms,
# Local KMS backend, per-key authorization enforced.
"e2e": lambda site: {**_local_kms(site), "RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY": "true"},
}
KMS_CONFIGS = ("e2", "e2e")
CONFIG_ORDER = ("e0", "e1x-text", "e1x-short", "e1", "e2", "e2e")
@dataclass
class Case:
case_id: str
title: str
level: str
layer: str
configs: tuple[str, ...]
fn: object
last: bool = False
CASES: list[Case] = []
def case(case_id: str, title: str, configs, level: str = "p0", layer: str = "L1", last: bool = False):
"""Register a case. `last` moves it to the end of its configuration, for cases that change process state."""
def register(fn):
CASES.append(Case(case_id, title, level, layer, tuple(configs), fn, last))
return fn
return register
# ---------------------------------------------------------------------------
# Case context
# ---------------------------------------------------------------------------
@dataclass
class Ctx:
site: h.Site
config: str
case_id: str
# False for a binary older than the `sse_local_master_key` startup event; the event is then recorded, not asserted.
master_key_event: bool = True
failures: list[str] = field(default_factory=list)
known: list[str] = field(default_factory=list)
fixed: list[str] = field(default_factory=list)
rows: list[dict] = field(default_factory=list)
_row: str = ""
# -- expectations ------------------------------------------------------
def known_deviation(self, holds: bool, expectation: str) -> None:
"""An expectation the product is known not to meet today.
It stays written down as the intended behavior. While it does not hold the case reports a known deviation
instead of a failure; once it holds the case says so, and the marker must then become a plain `check`.
"""
(self.fixed if holds else self.known).append(f"[{self._row}] {expectation}" if self._row else expectation)
def check(self, condition: bool, message: str) -> bool:
if not condition:
self.failures.append(f"[{self._row}] {message}" if self._row else message)
return bool(condition)
def need(self, condition: bool, message: str) -> None:
if not condition:
raise h.CheckFailed(message)
@contextlib.contextmanager
def row(self, name: str):
"""One input row of a table-driven case; its verdict is reported on its own."""
before = len(self.failures)
self._row = name
observe.set_row(name)
try:
yield
except h.CheckFailed as err:
self.failures.append(f"[{name}] {err}")
finally:
self.rows.append({"row": name, "verdict": FAIL if len(self.failures) > before else PASS,
"reasons": self.failures[before:]})
self._row = ""
observe.set_row("")
def ok(self, response, what: str, want: int = 200):
"""The exchange answered `want`; anything else ends the case, since later steps depend on it."""
status, headers, body = response
if status != want:
code, message = h.s3_error(body)
raise h.CheckFailed(f"{what}: HTTP {status} {code} {message[:200]!r}, expected {want}")
return headers, body
def refused(self, response, status: int, code: str, what: str, message_has: str | None = None) -> bool:
"""An explicit refusal: the exact status, an S3 error document, and the exact code."""
got_status, _, body = response
got_code, got_message = h.s3_error(body)
held = self.check(got_status == status and got_code == code,
f"{what}: HTTP {got_status} {got_code!r} {got_message[:200]!r}, expected {status} {code!r}")
if message_has is not None:
held &= self.check(message_has in got_message,
f"{what}: message {got_message[:240]!r} does not contain {message_has!r}")
return held
def projection(self, headers: dict[str, str], want: dict[str, str], what: str) -> None:
"""The SSE headers present are exactly `want`: nothing missing and nothing extra."""
got = h.projection(headers)
self.check(got == want, f"{what}: SSE headers {got}, expected exactly {want}")
def projection_with_known_extra(self, headers: dict[str, str], want: dict[str, str], extra: dict[str, str],
what: str) -> None:
"""The SSE headers are exactly `want`; today's product adds exactly `extra`, which is the known deviation.
Only that one documented difference is tolerated. Any other header set is a failure, so an unrelated
regression cannot hide behind the deviation.
"""
got = h.projection(headers)
if got == {**want, **extra}:
self.known_deviation(False, f"{what}: SSE headers are exactly {want}; today the response also carries {extra}")
elif self.check(got == want, f"{what}: SSE headers {got}, expected exactly {want} (known deviation: also {extra})"):
self.known_deviation(True, f"{what}: SSE headers are exactly {want}")
def no_sse_headers(self, headers: dict[str, str], what: str) -> None:
self.projection(headers, {}, what)
def bytes_equal(self, headers: dict[str, str], body: bytes, data: bytes, what: str) -> None:
self.check(h.TRUNCATED not in headers,
f"{what}: the server ended the body after {headers.get(h.TRUNCATED)} of {headers.get('content-length')} bytes")
self.check(h.sha256_hex(body) == h.sha256_hex(data),
f"{what}: body sha256 {h.sha256_hex(body)[:16]} len {len(body)}, expected {h.sha256_hex(data)[:16]} len {len(data)}")
self.check(headers.get("content-length") == str(len(data)),
f"{what}: Content-Length {headers.get('content-length')}, expected {len(data)}")
def absent(self, bucket: str, key: str, what: str, headers=None) -> None:
"""A refused write left nothing behind."""
status, _ = h.head(self.site, bucket, key, headers=headers, label="HeadObject(residue)")
self.check(status == 404, f"{what}: HEAD after the refused write is {status}, expected 404 (nothing may be stored)")
# -- common flows ------------------------------------------------------
def bucket(self, suffix: str = "") -> str:
name = f"bb-{self.case_id.lower()}" + (f"-{suffix}" if suffix else "")
h.make_bucket(self.site, name)
return name
def stored(self, bucket: str, key: str) -> list[str] | None:
"""Record which encryption metadata keys the object carries on disk."""
keys = h.stored_encryption_keys(self.site, bucket, key)
observe.note("stored-keys", f"{bucket}/{key}", keys)
return keys
def roundtrip(self, bucket: str, key: str, data: bytes, write_headers: dict | None, read_headers: dict | None,
want: dict[str, str], what: str) -> None:
"""PUT, GET and HEAD one object: the projection holds on all three and the bytes come back unchanged."""
put_headers, _ = self.ok(h.put(self.site, bucket, key, data, write_headers), f"{what} PUT")
self.projection(put_headers, want, f"{what} PUT")
get_headers, body = self.ok(h.get(self.site, bucket, key, read_headers), f"{what} GET")
self.projection(get_headers, want, f"{what} GET")
self.bytes_equal(get_headers, body, data, f"{what} GET")
status, head_headers = h.head(self.site, bucket, key, read_headers)
if self.check(status == 200, f"{what} HEAD: HTTP {status}, expected 200"):
self.projection(head_headers, want, f"{what} HEAD")
self.check(head_headers.get("content-length") == str(len(data)),
f"{what} HEAD: Content-Length {head_headers.get('content-length')}, expected {len(data)}")
self.stored(bucket, key)
def read_back(self, bucket: str, key: str, data: bytes, read_headers: dict | None, want: dict[str, str],
what: str) -> None:
get_headers, body = self.ok(h.get(self.site, bucket, key, read_headers), f"{what} GET")
self.projection(get_headers, want, f"{what} GET")
self.bytes_equal(get_headers, body, data, f"{what} GET")
def startup_event(self, event: str) -> list[dict]:
records = h.log_events(self.site, event)
fields = [{k: r.get(k) for k in ("level", "state", "enforced") if k in r} for r in records]
observe.note("startup-event", event, fields)
return records
-643
View File
@@ -1,643 +0,0 @@
"""Core library of the SSE black-box suite: one real server process, signed HTTP, and what a client can observe.
Everything a case does to the product goes through `signed_request`, which also records the exchange as an
observation (see observe.py). Standard library only, so the suite runs with `uv run --no-project`.
"""
from __future__ import annotations
import base64
import datetime
import hashlib
import hmac
import http.client
import json
import os
import re
import signal
import socket
import subprocess
import time
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from pathlib import Path
import observe
HERE = Path(__file__).resolve().parent
REPO_ROOT = HERE.parents[2]
STATE_DIR = Path(os.environ.get("SSE_BB_STATE_DIR", REPO_ROOT / "target" / "sse-bb"))
ADMIN_PREFIX = "/rustfs/admin/v3"
REGION = "us-east-1"
SERVICE = "s3"
UNSIGNED_PAYLOAD = "UNSIGNED-PAYLOAD"
# Added by the client to a response whose body ended before the announced length; never sent by the server.
TRUNCATED = "x-harness-body-truncated"
ACCESS_KEY = "rustfsadmin"
SECRET_KEY = "rustfsadmin"
# The port is a local-development choice; CI and other machines override it.
DEFAULT_PORT = int(os.environ.get("SSE_BB_PORT", "20105"))
STARTUP_TIMEOUT = 90.0
STOP_GRACE = 10.0
KMS_PROBE_INTERVAL = 5 # the product floors RUSTFS_KMS_PROBE_INTERVAL_SECS at 5 seconds
DEFAULT_KEY = "bb-default"
KEY_1 = "bb-key-1"
KEY_2 = "bb-key-2"
MISSING_KEY = "bb-key-that-was-never-created"
KIB = 1024
MIB = 1024 * 1024
class CheckFailed(Exception):
"""A hard prerequisite of a case does not hold; the case cannot continue."""
class EnvError(Exception):
"""The environment, not the product, is at fault: the site is down or a dependency is missing."""
# ---------------------------------------------------------------------------
# Deterministic test data
# ---------------------------------------------------------------------------
def plain(seed: str, size: int) -> bytes:
"""Incompressible bytes fully determined by `seed`, so two runs expect the same body hashes."""
out = bytearray()
counter = 0
while len(out) < size:
out += hashlib.sha256(f"sse-blackbox plaintext {seed} {counter}".encode()).digest()
counter += 1
return bytes(out[:size])
def derived_key(label: str, size: int = 32) -> bytes:
"""Key material derived at run time; the repository carries no key literal."""
return hashlib.sha256(f"sse-blackbox key {label}".encode()).digest()[:size]
def sha256_hex(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
# ---------------------------------------------------------------------------
# Site
# ---------------------------------------------------------------------------
@dataclass
class Site:
name: str
root: Path
port: int = DEFAULT_PORT
access_key: str = ACCESS_KEY
secret_key: str = SECRET_KEY
host: str = "127.0.0.1"
env: dict = field(default_factory=dict)
proc: subprocess.Popen | None = None
@property
def endpoint(self) -> str:
return f"http://{self.host}:{self.port}"
@property
def volume_dir(self) -> Path:
return self.root / "volume"
@property
def log_dir(self) -> Path:
return self.root / "logs"
@property
def stdout_log(self) -> Path:
return self.log_dir / "stdout.log"
@property
def app_log(self) -> Path:
"""JSON-lines application log written under RUSTFS_OBS_LOG_DIRECTORY."""
return self.log_dir / "rustfs.log"
@property
def key_dir(self) -> Path:
return self.root / "kms-keys"
def drives(self) -> list[Path]:
return [self.volume_dir / f"test{i}" for i in range(1, 5)]
# ---------------------------------------------------------------------------
# SigV4 and HTTP
# ---------------------------------------------------------------------------
def _hmac(key: bytes, msg: str) -> bytes:
return hmac.new(key, msg.encode(), hashlib.sha256).digest()
def _uri_encode(value: str, encode_slash: bool) -> str:
return urllib.parse.quote(value, safe="-._~" + ("" if encode_slash else "/"))
def _canonical_query(query: str) -> str:
if not query:
return ""
pairs = urllib.parse.parse_qsl(query, keep_blank_values=True)
return "&".join(f"{k}={v}" for k, v in sorted((_uri_encode(k, True), _uri_encode(v, True)) for k, v in pairs))
def _signing_key(secret_key: str, date_stamp: str) -> bytes:
return _hmac(_hmac(_hmac(_hmac(f"AWS4{secret_key}".encode(), date_stamp), REGION), SERVICE), "aws4_request")
def _http(url: str, method: str, body: bytes | None, headers: dict[str, str], timeout: float):
request = urllib.request.Request(url, data=body, method=method)
for name, value in headers.items():
if name != "host":
request.add_header(name, value)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
response_headers = {k.lower(): v for k, v in response.headers.items()}
try:
return response.status, response_headers, response.read()
except http.client.IncompleteRead as err:
# The server answered and then cut the body short. That is the product's doing, not the
# environment's, so the status stands and the truncation is reported next to it.
return response.status, {**response_headers, TRUNCATED: str(len(err.partial))}, err.partial
except urllib.error.HTTPError as err:
return err.code, {k.lower(): v for k, v in err.headers.items()}, err.read()
except (urllib.error.URLError, OSError, TimeoutError, http.client.HTTPException) as err:
# Status 0 is a transport failure: the site is down or hung. It is never a product refusal.
return 0, {}, str(err).encode()
def signed_request(
s: Site,
method: str,
path: str,
query: str = "",
body: bytes | None = None,
headers: dict[str, str] | None = None,
creds: tuple[str, str] | None = None,
timeout: float = 120.0,
label: str | None = None,
) -> tuple[int, dict[str, str], bytes]:
"""SigV4 header-signed request. Returns (status, lower-cased headers, body) and never raises on HTTP errors."""
access_key, secret_key = creds if creds else (s.access_key, s.secret_key)
now = datetime.datetime.now(datetime.UTC)
amz_date = now.strftime("%Y%m%dT%H%M%SZ")
date_stamp = now.strftime("%Y%m%d")
all_headers = {"host": f"{s.host}:{s.port}", "x-amz-content-sha256": UNSIGNED_PAYLOAD, "x-amz-date": amz_date}
if headers:
all_headers.update({k.lower(): v for k, v in headers.items()})
signed_names = ";".join(sorted(all_headers))
canonical_headers = "".join(f"{k}:{all_headers[k].strip()}\n" for k in sorted(all_headers))
canonical_request = "\n".join(
[method, _uri_encode(path, False), _canonical_query(query), canonical_headers, signed_names, UNSIGNED_PAYLOAD]
)
scope = f"{date_stamp}/{REGION}/{SERVICE}/aws4_request"
string_to_sign = "\n".join(["AWS4-HMAC-SHA256", amz_date, scope, sha256_hex(canonical_request.encode())])
signature = hmac.new(_signing_key(secret_key, date_stamp), string_to_sign.encode(), hashlib.sha256).hexdigest()
all_headers["authorization"] = (
f"AWS4-HMAC-SHA256 Credential={access_key}/{scope}, SignedHeaders={signed_names}, Signature={signature}"
)
url = f"{s.endpoint}{urllib.parse.quote(path, safe='/-._~')}"
if query:
url += f"?{query}"
status, response_headers, response_body = _http(url, method, body, all_headers, timeout)
observe.exchange(label, method, path, query, status, response_headers, response_body)
if status == 0 and observe.in_case():
raise EnvError(f"{method} {path}: no HTTP response ({response_body[:160].decode(errors='replace')})")
return status, response_headers, response_body
def admin(s: Site, method: str, subpath: str, query: str = "", payload: object | None = None,
creds: tuple[str, str] | None = None, raw_body: bytes | None = None,
label: str | None = None) -> tuple[int, object]:
"""Signed admin call; returns (status, parsed JSON or text)."""
body = json.dumps(payload).encode() if payload is not None else raw_body
headers = {"content-type": "application/json"} if body is not None else None
status, _, raw = signed_request(s, method, f"{ADMIN_PREFIX}/{subpath.lstrip('/')}", query, body, headers, creds,
label=label)
try:
return status, json.loads(raw) if raw else None
except json.JSONDecodeError:
return status, raw.decode(errors="replace")
def s3_error(body: bytes) -> tuple[str, str]:
"""(Code, Message) from an S3 XML error body; ('', '') when the body is not such a document."""
try:
root = ET.fromstring(body)
except ET.ParseError:
return "", ""
return root.findtext("{*}Code") or root.findtext("Code") or "", root.findtext("{*}Message") or root.findtext("Message") or ""
# ---------------------------------------------------------------------------
# SSE request headers and response projections
# ---------------------------------------------------------------------------
SSE = "x-amz-server-side-encryption"
SSE_KEY_ID = f"{SSE}-aws-kms-key-id"
SSE_CONTEXT = f"{SSE}-context"
SSEC = f"{SSE}-customer-"
COPY_SSEC = "x-amz-copy-source-server-side-encryption-customer-"
def sse_s3() -> dict[str, str]:
return {SSE: "AES256"}
def sse_kms(key_id: str | None = None, context: dict | None = None) -> dict[str, str]:
headers = {SSE: "aws:kms"}
if key_id:
headers[SSE_KEY_ID] = key_id
if context is not None:
headers[SSE_CONTEXT] = base64.b64encode(json.dumps(context, separators=(",", ":")).encode()).decode()
return headers
def key_md5(key: bytes) -> str:
return base64.b64encode(hashlib.md5(key).digest()).decode()
def ssec(key: bytes, prefix: str = SSEC) -> dict[str, str]:
"""The SSE-C triple. Pass `COPY_SSEC` as the prefix for the copy-source variant."""
return {f"{prefix}algorithm": "AES256", f"{prefix}key": base64.b64encode(key).decode(), f"{prefix}key-md5": key_md5(key)}
def projection(headers: dict[str, str]) -> dict[str, str]:
"""The SSE headers actually present in a response."""
return {k: v for k, v in headers.items() if k.startswith(SSE)}
def projection_s3() -> dict[str, str]:
return {SSE: "AES256"}
def projection_kms(key_id: str) -> dict[str, str]:
return {SSE: "aws:kms", SSE_KEY_ID: key_id}
def projection_c(key: bytes) -> dict[str, str]:
return {f"{SSEC}algorithm": "AES256", f"{SSEC}key-md5": key_md5(key)}
# ---------------------------------------------------------------------------
# S3 operations
# ---------------------------------------------------------------------------
def make_bucket(s: Site, bucket: str) -> None:
"""Create a bucket, riding out the few seconds after startup in which a fresh site still answers 503."""
deadline = time.monotonic() + 30.0
with observe.quiet():
while True:
status, _, body = signed_request(s, "PUT", f"/{bucket}")
if status in (200, 409) or (0 < status < 500) or time.monotonic() > deadline:
break
time.sleep(0.5)
if status == 0:
raise EnvError(f"CreateBucket {bucket}: no HTTP response")
if status not in (200, 409):
raise CheckFailed(f"CreateBucket {bucket}: HTTP {status} {s3_error(body)}")
def put(s: Site, bucket: str, key: str, data: bytes, headers=None, creds=None, label=None):
return signed_request(s, "PUT", f"/{bucket}/{key}", body=data,
headers={"content-type": "application/octet-stream", **(headers or {})}, creds=creds,
label=label or "PutObject")
def get(s: Site, bucket: str, key: str, headers=None, creds=None, range_: str | None = None, label=None):
request_headers = dict(headers or {})
if range_:
request_headers["range"] = range_
return signed_request(s, "GET", f"/{bucket}/{key}", headers=request_headers or None, creds=creds,
label=label or "GetObject")
def head(s: Site, bucket: str, key: str, headers=None, creds=None, label=None):
status, response_headers, _ = signed_request(s, "HEAD", f"/{bucket}/{key}", headers=headers, creds=creds,
label=label or "HeadObject")
return status, response_headers
def copy(s: Site, bucket: str, key: str, src_bucket: str, src_key: str, headers=None, creds=None, label=None):
"""CopyObject. `headers` carries destination SSE headers and the copy-source SSE-C triple."""
request_headers = {"x-amz-copy-source": f"/{src_bucket}/{urllib.parse.quote(src_key)}", **(headers or {})}
return signed_request(s, "PUT", f"/{bucket}/{key}", headers=request_headers, creds=creds, label=label or "CopyObject")
def create_multipart(s: Site, bucket: str, key: str, headers=None, creds=None):
status, response_headers, body = signed_request(s, "POST", f"/{bucket}/{key}", query="uploads=", headers=headers,
creds=creds, label="CreateMultipartUpload")
upload_id = None
if status == 200:
upload_id = ET.fromstring(body).findtext("{*}UploadId") or ET.fromstring(body).findtext("UploadId")
return status, response_headers, upload_id, body
def upload_part(s: Site, bucket: str, key: str, upload_id: str, part_no: int, data: bytes, headers=None, creds=None):
return signed_request(s, "PUT", f"/{bucket}/{key}", query=f"partNumber={part_no}&uploadId={urllib.parse.quote(upload_id)}",
body=data, headers=headers, creds=creds, label="UploadPart")
def complete_multipart(s: Site, bucket: str, key: str, upload_id: str, parts: list[tuple[int, str]], headers=None,
creds=None):
parts_xml = "".join(f"<Part><PartNumber>{n}</PartNumber><ETag>{e}</ETag></Part>" for n, e in parts)
return signed_request(s, "POST", f"/{bucket}/{key}", query=f"uploadId={urllib.parse.quote(upload_id)}",
body=f"<CompleteMultipartUpload>{parts_xml}</CompleteMultipartUpload>".encode(),
headers={"content-type": "application/xml", **(headers or {})}, creds=creds,
label="CompleteMultipartUpload")
def list_parts(s: Site, bucket: str, key: str, upload_id: str):
return signed_request(s, "GET", f"/{bucket}/{key}", query=f"uploadId={urllib.parse.quote(upload_id)}", label="ListParts")
def list_objects(s: Site, bucket: str, prefix: str = "") -> dict[str, int]:
"""{key: size} from ListObjectsV2."""
query = "list-type=2" + (f"&prefix={urllib.parse.quote(prefix)}" if prefix else "")
status, _, body = signed_request(s, "GET", f"/{bucket}", query=query, label="ListObjectsV2")
if status != 200:
raise CheckFailed(f"ListObjectsV2 {bucket}: HTTP {status} {s3_error(body)}")
out = {}
for contents in ET.fromstring(body).findall("{*}Contents"):
out[contents.findtext("{*}Key")] = int(contents.findtext("{*}Size"))
return out
def put_bucket_encryption(s: Site, bucket: str, algorithm: str, kms_key_id: str | None = None):
key_xml = f"<KMSMasterKeyID>{kms_key_id}</KMSMasterKeyID>" if kms_key_id else ""
payload = ('<ServerSideEncryptionConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule>'
f"<ApplyServerSideEncryptionByDefault><SSEAlgorithm>{algorithm}</SSEAlgorithm>{key_xml}"
"</ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>").encode()
return signed_request(s, "PUT", f"/{bucket}", query="encryption=", body=payload,
headers={"content-type": "application/xml"}, label="PutBucketEncryption")
def get_bucket_encryption(s: Site, bucket: str) -> tuple[int, dict, bytes]:
"""(status, {'algorithm', 'kms_key_id'}, body)."""
status, _, body = signed_request(s, "GET", f"/{bucket}", query="encryption=", label="GetBucketEncryption")
config = {}
if status == 200:
root = ET.fromstring(body)
config = {"algorithm": root.findtext(".//{*}SSEAlgorithm"), "kms_key_id": root.findtext(".//{*}KMSMasterKeyID")}
return status, config, body
def delete_bucket_encryption(s: Site, bucket: str):
return signed_request(s, "DELETE", f"/{bucket}", query="encryption=", label="DeleteBucketEncryption")
# ---------------------------------------------------------------------------
# Admin: KMS and IAM
# ---------------------------------------------------------------------------
def kms_status(s: Site):
return admin(s, "GET", "kms/status", label="KmsStatus")
def kms_stop(s: Site):
return admin(s, "POST", "kms/stop", label="KmsStop")
def kms_start(s: Site):
return admin(s, "POST", "kms/start", payload={}, label="KmsStart")
def key_create(s: Site, name: str):
return admin(s, "POST", "kms/keys", payload={"tags": {"name": name}}, label="KmsCreateKey")
def key_list(s: Site) -> list[str]:
status, response = admin(s, "GET", "kms/keys", query="limit=100", label="KmsListKeys")
if status != 200 or not isinstance(response, dict):
raise CheckFailed(f"ListKeys: HTTP {status} {str(response)[:200]}")
return sorted(str(k.get("key_id")) for k in response.get("keys", []))
def write_local_key_file(key_dir: Path, key_id: str) -> None:
"""Pre-create a Local-backend key file in the development format, with material generated for this run."""
key_dir.mkdir(parents=True, exist_ok=True)
created = datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds").replace("+00:00", "Z")
stored = {
"key_id": key_id, "version": 1, "algorithm": "AES_256", "usage": "EncryptDecrypt", "status": "Active",
"metadata": {}, "created_at": f"{created}[UTC]", "rotated_at": None, "created_by": "sse-blackbox",
"encrypted_key_material": base64.b64encode(os.urandom(32)).decode(), "nonce": [],
}
path = key_dir / f"{key_id}.key"
path.write_text(json.dumps(stored, indent=2))
path.chmod(0o600)
def wait_kms_healthy(s: Site, timeout: float = 60.0) -> None:
deadline = time.monotonic() + timeout
with observe.quiet():
while True:
status, body = kms_status(s)
if status == 200 and isinstance(body, dict) and body.get("backend_status") == "healthy":
return
if time.monotonic() > deadline:
raise EnvError(f"KMS not healthy within {timeout:.0f}s: HTTP {status} {str(body)[:200]}")
time.sleep(0.5)
def ensure_key(s: Site, name: str) -> None:
"""Create the named key unless the key list already has it."""
with observe.quiet():
if name in key_list(s):
return
status, response = key_create(s, name)
if status not in (200, 201):
raise EnvError(f"CreateKey {name}: HTTP {status} {str(response)[:200]}")
def iam_add_user(s: Site, user: str, secret: str):
return admin(s, "PUT", "add-user", query=f"accessKey={urllib.parse.quote(user)}",
payload={"secretKey": secret, "status": "enabled"}, label="AddUser")
def iam_add_policy(s: Site, name: str, policy: dict):
return admin(s, "PUT", "add-canned-policy", query=f"name={urllib.parse.quote(name)}",
raw_body=json.dumps(policy).encode(), label="AddPolicy")
def iam_attach_policy(s: Site, user: str, policy: str):
query = f"policyName={urllib.parse.quote(policy)}&userOrGroup={urllib.parse.quote(user)}&isGroup=false"
return admin(s, "PUT", "set-user-or-group-policy", query=query, label="AttachPolicy")
# ---------------------------------------------------------------------------
# Process control
# ---------------------------------------------------------------------------
BASE_SERVER_ENV = {
"RUST_LOG": "rustfs=info,rustfs_sse=info,rustfs_kms=info,rustfs_iam=info,rustfs_ecstore=warn,rustfs_scanner=warn,rustfs_heal=warn,s3s=warn",
"RUSTFS_CONSOLE_ENABLE": "false",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
}
PROXY_VARS = ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "ALL_PROXY", "all_proxy")
SECRET_ENV_MARKERS = ("TOKEN", "SECRET", "MASTER_KEY", "PASSWORD")
def clean_env() -> dict[str, str]:
"""The inherited environment without RUSTFS_* and proxy variables: ambient configuration must not reach a run."""
env = {k: v for k, v in os.environ.items() if not k.startswith("RUSTFS_") and k not in PROXY_VARS}
env["NO_PROXY"] = env["no_proxy"] = "127.0.0.1,localhost"
return env
def port_in_use(port: int) -> bool:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.settimeout(0.5)
return sock.connect_ex(("127.0.0.1", port)) == 0
def is_running(s: Site) -> bool:
return s.proc is not None and s.proc.poll() is None
def start_site(s: Site, binary: Path, env_overrides: dict[str, str]) -> None:
"""Start `s` on a scrubbed environment. A busy port is an error: the suite neither moves on nor kills the holder."""
if is_running(s):
raise EnvError(f"{s.name} is already running")
if port_in_use(s.port):
raise EnvError(f"port {s.port} is already in use; find the holder with: lsof -nP -iTCP:{s.port} -sTCP:LISTEN")
if not binary.exists():
raise EnvError(f"server binary missing: {binary}")
for drive in s.drives():
drive.mkdir(parents=True, exist_ok=True)
s.log_dir.mkdir(parents=True, exist_ok=True)
env = clean_env()
env.update(BASE_SERVER_ENV)
env.update({
"RUSTFS_ACCESS_KEY": s.access_key,
"RUSTFS_SECRET_KEY": s.secret_key,
"RUSTFS_VOLUMES": str(s.volume_dir / "test{1...4}"),
"RUSTFS_ADDRESS": f"127.0.0.1:{s.port}",
"RUSTFS_SERVER_DOMAINS": f"127.0.0.1:{s.port}",
"RUSTFS_OBS_LOG_DIRECTORY": str(s.log_dir),
})
env.update(env_overrides)
s.env = dict(env_overrides)
shown = {k: ("<redacted>" if any(m in k for m in SECRET_ENV_MARKERS) else v) for k, v in s.env.items()}
with s.stdout_log.open("ab") as log:
log.write(f"\n===== start {datetime.datetime.now().isoformat(timespec='seconds')} env={json.dumps(shown)}\n".encode())
s.proc = subprocess.Popen([str(binary)], cwd=str(s.root), env=env, stdout=log, stderr=subprocess.STDOUT,
start_new_session=True)
(s.log_dir / "rustfs.pid").write_text(f"{s.proc.pid}\n")
wait_ready(s)
def wait_ready(s: Site, timeout: float = STARTUP_TIMEOUT) -> None:
"""Ready means a signed ListBuckets answers; a listening port alone proves nothing."""
deadline = time.monotonic() + timeout
with observe.quiet():
while True:
if not is_running(s):
raise EnvError(f"{s.name} exited during startup; log tail:\n{tail_log(s)}")
status, _, _ = signed_request(s, "GET", "/", timeout=3.0)
if 200 <= status < 500:
return
if time.monotonic() > deadline:
raise EnvError(f"{s.name} not ready within {timeout:.0f}s; log tail:\n{tail_log(s)}")
time.sleep(0.5)
def stop_site(s: Site) -> None:
"""SIGTERM, then SIGKILL after a grace period. Only the process this driver started is ever signalled."""
if s.proc is None:
return
if s.proc.poll() is None:
s.proc.send_signal(signal.SIGTERM)
try:
s.proc.wait(timeout=STOP_GRACE)
except subprocess.TimeoutExpired:
s.proc.kill()
s.proc.wait(timeout=STOP_GRACE)
s.proc = None
deadline = time.monotonic() + 30.0
while port_in_use(s.port):
if time.monotonic() > deadline:
raise EnvError(f"{s.name} still listening on {s.port} after stop")
time.sleep(0.2)
def tail_log(s: Site, lines: int = 15) -> str:
out = []
for path in (s.stdout_log, s.app_log):
try:
text = path.read_text(errors="replace")
except FileNotFoundError:
continue
out.append(f"--- {path.name} ---\n" + "\n".join(line[:400] for line in text.splitlines()[-lines:]))
return "\n".join(out) or "<no log>"
def log_events(s: Site, event: str) -> list[dict]:
"""Structured log records whose `event` field equals `event`, in the order they were written."""
records = []
try:
text = s.app_log.read_text(errors="replace")
except FileNotFoundError:
return records
for line in text.splitlines():
if f'"{event}"' not in line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
fields = record.get("fields") if isinstance(record.get("fields"), dict) else {}
merged = {**record, **fields}
if merged.get("event") == event:
records.append(merged)
return records
def log_text(s: Site) -> str:
return "".join(p.read_text(errors="replace") for p in (s.stdout_log, s.app_log) if p.exists())
# ---------------------------------------------------------------------------
# Stored form (layer 2): metadata key names, read from the raw xl.meta
# ---------------------------------------------------------------------------
_STORED_KEY = re.compile(
rb"(?i)(x-rustfs-encryption-[a-z0-9-]+|x-rustfs-internal-[a-z0-9-]+|x-minio-internal-[a-z0-9-]+"
rb"|x-amz-server-side-encryption[a-z0-9-]*)"
)
def stored_encryption_keys(s: Site, bucket: str, key: str) -> list[str] | None:
"""Encryption-related metadata key names stored for an object, lower-cased and sorted; None when not found.
Key names are plain strings inside xl.meta, so a byte scan finds them without a format-aware reader. It reads
names only; values need the dedicated layer-2 tooling.
"""
for drive in s.drives():
xl = drive / bucket / key / "xl.meta"
if xl.exists():
return sorted({m.group(0).decode().lower() for m in _STORED_KEY.finditer(xl.read_bytes())})
return None
def binary_identity(binary: Path) -> dict:
try:
version = subprocess.run([str(binary), "--version"], capture_output=True, text=True, timeout=30, check=False,
env=clean_env()).stdout.strip()
except (OSError, subprocess.SubprocessError) as err:
version = f"<error {err}>"
return {"path": str(binary), "sha256": sha256_hex(binary.read_bytes()), "version": version.splitlines()[0] if version else ""}
-205
View File
@@ -1,205 +0,0 @@
"""Observation vectors: everything a client can see in one exchange, normalized so two runs compare field by field.
Each exchange inside a case becomes one record keyed by (config, case, seq). Two binaries driven by the same case
issue the same request sequence, so equal keys denote the same request and any field that differs is a behavior
difference. Values that are random by design (version ids, upload ids, ETags of encrypted objects) are reduced to
their shape.
"""
from __future__ import annotations
import contextlib
import hashlib
import json
import re
import xml.etree.ElementTree as ET
_records: list[dict] = []
_context: dict | None = None
_quiet = 0
# Response headers whose values are deterministic for a deterministic request and therefore compared verbatim.
_VERBATIM = ("content-range", "content-type", "x-amz-mp-parts-count", "x-amz-storage-class", "x-amz-delete-marker")
_HEX32 = re.compile(r'^"?[0-9a-f]{32}"?$')
_HEX32_PARTS = re.compile(r'^"?[0-9a-f]{32}-(\d+)"?$')
_UUID = re.compile(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}")
_LONG_TOKEN = re.compile(r"[A-Za-z0-9+/_=-]{40,}")
_RANDOM_QUERY = ("uploadId", "versionId")
# Elements of a result document whose value differs from run to run; only their presence is compared.
_XML_VOLATILE = frozenset({
"UploadId", "LastModified", "Initiated", "VersionId", "RequestId", "HostId", "UploadIdMarker",
"NextUploadIdMarker", "VersionIdMarker", "NextVersionIdMarker", "ContinuationToken", "NextContinuationToken",
})
_ENDPOINT = re.compile(r"^https?://[^/]+")
# Headers that leak storage internals if they ever reach a client.
_INTERNAL_PREFIXES = ("x-rustfs-encryption-", "x-rustfs-internal-", "x-minio-internal-")
_INTERNAL_EXACT = ("x-amz-server-side-encryption-customer-key",)
def begin_case(config: str, case: str) -> None:
global _context
_context = {"config": config, "case": case, "seq": 0, "row": ""}
def end_case() -> None:
global _context
_context = None
def in_case() -> bool:
return _context is not None and _quiet == 0
def set_row(row: str) -> None:
if _context is not None:
_context["row"] = row
@contextlib.contextmanager
def quiet():
"""Suspend recording around polling loops, whose iteration count is timing and not behavior."""
global _quiet
_quiet += 1
try:
yield
finally:
_quiet -= 1
def drain() -> list[dict]:
out = list(_records)
_records.clear()
return out
def _etag_shape(value: str | None) -> str | None:
if value is None:
return None
if _HEX32.match(value):
return "hex32"
parts = _HEX32_PARTS.match(value)
if parts:
return f"hex32-{parts.group(1)}"
return "other"
def normalize_message(message: str) -> str:
"""Strip the parts of an error message that are random per request, keeping the wording."""
return _LONG_TOKEN.sub("<token>", _UUID.sub("<uuid>", message))
def _query_shape(query: str) -> str:
parts = []
for pair in query.split("&") if query else []:
name = pair.partition("=")[0]
parts.append(f"{name}=<id>" if name in _RANDOM_QUERY else pair)
return "&".join(parts)
def _xml_leaves(root: ET.Element) -> list[list[str]]:
"""Every leaf of a result document as [path, value], in document order.
Order and repetition are kept, so a listing with a missing, extra or reordered entry differs. Values are kept
verbatim except for the elements known to vary between runs.
"""
leaves: list[list[str]] = []
def walk(element: ET.Element, path: str) -> None:
name = element.tag.rpartition("}")[2]
here = f"{path}/{name}" if path else name
children = list(element)
if children:
for child in children:
walk(child, here)
return
text = (element.text or "").strip()
if name in _XML_VOLATILE:
value = "<present>" if text else ""
elif name == "ETag":
value = _etag_shape(text) or ""
elif name == "Location":
value = _ENDPOINT.sub("<endpoint>", text)
else:
value = text
leaves.append([here, value])
walk(root, "")
return leaves
def _body_shape(status: int, method: str, headers: dict[str, str], body: bytes) -> dict:
"""What the body says, without its random parts."""
if not body:
return {"kind": "empty"}
# An object body is opaque even when its first byte happens to look like a document.
is_object = method == "GET" and headers.get("content-type", "").startswith("application/octet-stream")
text_start = b"" if is_object else body[:64].lstrip()
if text_start.startswith(b"<"):
try:
root = ET.fromstring(body)
except ET.ParseError:
return {"kind": "xml-unparsed", "length": len(body)}
tag = root.tag.rpartition("}")[2]
if tag == "Error":
code = root.findtext("{*}Code") or root.findtext("Code") or ""
message = root.findtext("{*}Message") or root.findtext("Message") or ""
return {"kind": "s3-error", "code": code, "message": normalize_message(message)}
return {"kind": "xml", "root": tag, "leaves": _xml_leaves(root)}
if text_start.startswith((b"{", b"[")):
try:
parsed = json.loads(body)
except ValueError: # not JSON, or not text at all
parsed = None
if isinstance(parsed, dict):
shape = {"kind": "json", "keys": sorted(parsed)}
for name in ("success", "status", "code", "error", "message", "backend_type", "backend_status"):
if isinstance(parsed.get(name), (str, bool, int)) or parsed.get(name) is None and name in parsed:
value = parsed.get(name)
shape[name] = normalize_message(value) if isinstance(value, str) else value
return shape
if parsed is not None:
return {"kind": "json", "type": type(parsed).__name__}
# Object bodies: the plaintext is seed-determined, so the hash must match across runs.
return {"kind": "bytes", "length": len(body), "sha256": hashlib.sha256(body).hexdigest()}
def exchange(label: str | None, method: str, path: str, query: str, status: int, headers: dict[str, str],
body: bytes) -> None:
if not in_case():
return
_context["seq"] += 1
sse = {k: v for k, v in headers.items() if k.startswith("x-amz-server-side-encryption")}
internal = sorted(k for k in headers if k.startswith(_INTERNAL_PREFIXES) or k in _INTERNAL_EXACT)
record = {
"config": _context["config"],
"case": _context["case"],
"seq": _context["seq"],
"row": _context["row"],
"kind": "exchange",
"op": label or method,
"request": f"{method} {path}" + (f"?{_query_shape(query)}" if query else ""),
"status": status,
"sse": dict(sorted(sse.items())),
"headers": {name: headers[name] for name in _VERBATIM if name in headers},
"header_names": sorted(headers),
"etag": _etag_shape(headers.get("etag")),
"version_id": "present" if "x-amz-version-id" in headers else None,
"internal_headers": internal,
"body": _body_shape(status, method, headers, body),
}
# A HEAD response states the object length without sending a body, and object lengths are deterministic.
# Documents (errors, listings) carry request ids and timestamps, so their length is not recorded.
if method == "HEAD" or (method == "GET" and status in (200, 206) and record["body"]["kind"] in ("bytes", "empty")):
record["content_length"] = headers.get("content-length")
if "x-harness-body-truncated" in headers:
record["truncated_after"] = headers["x-harness-body-truncated"]
record["header_names"].remove("x-harness-body-truncated")
_records.append(record)
def note(kind: str, name: str, value) -> None:
"""Record a non-HTTP observation: a startup event, a stored key set, an admin fact."""
if _context is None:
return
_records.append({"config": _context["config"], "case": _context["case"], "seq": 0, "row": _context["row"],
"kind": kind, "name": name, "value": value})
-102
View File
@@ -1,102 +0,0 @@
"""Offline checks of the suite's own judgment: what it must tell apart, and what it must not accept.
They need no server. Each one states a pair the suite has to distinguish or a difference it has to ignore, so a
change that makes the suite blind in one of these ways fails here.
"""
from __future__ import annotations
import diffing
import framework as fw
import harness as h
import observe
LIST_PARTS = (
'<ListPartsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>b</Bucket><Key>k</Key>'
"<UploadId>{upload}</UploadId>{parts}</ListPartsResult>"
)
PART = '<Part><PartNumber>{number}</PartNumber><LastModified>{modified}</LastModified><ETag>"{etag}"</ETag><Size>{size}</Size></Part>'
def _observe_xml(body: str) -> dict:
observe.begin_case("selftest", "XML")
try:
observe.exchange("ListParts", "GET", "/b/k", "uploadId=x", 200, {"content-type": "application/xml"}, body.encode())
finally:
observe.end_case()
return observe.drain()[0]["body"]
def _listing(upload: str, modified: str, etag: str, sizes: list[int]) -> dict:
parts = "".join(PART.format(number=i, modified=modified, etag=etag, size=size) for i, size in enumerate(sizes, start=1))
return _observe_xml(LIST_PARTS.format(upload=upload, parts=parts))
def _xml_checks() -> list[str]:
failures = []
base = _listing("upload-a", "2026-01-01T00:00:00Z", "a" * 32, [5242880, 17])
if _listing("upload-b", "2026-02-02T00:00:00Z", "b" * 32, [5242880, 17]) != base:
failures.append("XML observation: a different upload id, timestamp or ETag value changes the observation")
if _listing("upload-a", "2026-01-01T00:00:00Z", "a" * 32, [1, 17]) == base:
failures.append("XML observation: a different part Size does not change the observation")
if _listing("upload-a", "2026-01-01T00:00:00Z", "a" * 32, [5242880]) == base:
failures.append("XML observation: a missing Part does not change the observation")
if _listing("upload-a", "2026-01-01T00:00:00Z", "a" * 32, [17, 5242880]) == base:
failures.append("XML observation: reordered parts do not change the observation")
return failures
def _verdict(headers: dict[str, str]) -> tuple[int, int, int]:
"""(failures, known deviations, fixed deviations) for an SSE-C multipart response with these SSE headers."""
key = h.derived_key("selftest")
ctx = fw.Ctx(site=None, config="selftest", case_id="SELFTEST")
ctx.projection_with_known_extra(headers, h.projection_c(key), {h.SSE: "AES256"}, "response")
return len(ctx.failures), len(ctx.known), len(ctx.fixed)
def _deviation_checks() -> list[str]:
failures = []
key = h.derived_key("selftest")
want = h.projection_c(key)
triple = h.ssec(key)
cases = {
"the intended headers": (want, (0, 0, 1)),
"the documented extra AES256 header": ({**want, h.SSE: "AES256"}, (0, 1, 0)),
"the raw customer key echoed back": ({**want, h.SSE: "AES256", f"{h.SSEC}key": triple[f"{h.SSEC}key"]}, (1, 0, 0)),
"an extra aws:kms header": ({**want, h.SSE: "aws:kms"}, (1, 0, 0)),
"a missing key MD5": ({f"{h.SSEC}algorithm": "AES256", h.SSE: "AES256"}, (1, 0, 0)),
}
for name, (headers, expected) in cases.items():
got = _verdict(headers)
if got != expected:
failures.append(f"known deviation: {name} gives (failures, known, fixed) = {got}, expected {expected}")
return failures
def _run(binary: dict, value) -> dict:
result = {"config": "e1", "case": "KS-02", "row": "", "verdict": fw.PASS}
note = {"config": "e1", "case": "KS-02", "seq": 0, "row": "", "kind": "startup-event", "name": "sse_local_master_key",
"value": value}
return {"dir": None, "meta": {"binary": binary}, "results": [result], "observations": [note]}
def _diff_checks() -> list[str]:
failures = []
old, new = {"master_key_event": False}, {"master_key_event": True}
loaded, not_set = [{"level": "INFO", "state": "loaded"}], [{"level": "INFO", "state": "not_set"}]
outcome = diffing.compare(_run(old, []), _run(new, loaded))
if outcome["blocking"] or len(outcome["allowed"]) != 1:
failures.append("diff: the event missing on a binary that predates it is not an accepted difference")
if not diffing.compare(_run(new, loaded), _run(new, not_set))["blocking"]:
failures.append("diff: two binaries that both have the event may disagree on its state")
if not diffing.compare(_run(new, []), _run(new, loaded))["blocking"]:
failures.append("diff: a binary that should have the event may lack it")
changed = _run(new, loaded)
changed["results"][0]["verdict"] = fw.FAIL
if not diffing.compare(_run(new, loaded), changed)["blocking"]:
failures.append("diff: a changed verdict is not reported")
return failures
def run() -> list[str]:
return [*_xml_checks(), *_deviation_checks(), *_diff_checks()]
-422
View File
@@ -1,422 +0,0 @@
#!/usr/bin/env python3
"""SSE black-box suite: register server binaries, run the cases against one, and compare two runs.
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py bin add --name head-default --path target/debug/rustfs --commit <sha> --build default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py run --binary head-default
uv run --python 3.12 --no-project scripts/test/sse_blackbox/suite.py diff <run-a> <run-b>
"""
from __future__ import annotations
import argparse
import atexit
import datetime
import fcntl
import json
import shutil
import sys
import time
import traceback
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import cases_copy_multipart # noqa: F401
import cases_negative_authz # noqa: F401
import cases_roundtrip # noqa: F401
import cases_state # noqa: F401
import diffing
import framework as fw
import harness as h
import observe
import selftest
BIN_DIR = h.STATE_DIR / "bin"
RUNS_DIR = h.STATE_DIR / "runs"
KEEP_RUNS = 5
# ---------------------------------------------------------------------------
# Binary registry
# ---------------------------------------------------------------------------
def cmd_bin_add(args) -> int:
source = Path(args.path).resolve()
if not source.is_file():
raise SystemExit(f"no such binary: {source}")
target_dir = BIN_DIR / args.name
target_dir.mkdir(parents=True, exist_ok=True)
target = target_dir / "rustfs"
shutil.copy2(source, target)
identity = h.binary_identity(target)
identity.update({"name": args.name, "commit": args.commit, "build": args.build, "source": str(source),
"master_key_event": not args.no_master_key_event})
(target_dir / "identity.json").write_text(json.dumps(identity, indent=1) + "\n")
print(f"registered {args.name}: {identity['version']} sha256 {identity['sha256'][:12]} build {args.build}")
return 0
def cmd_bin_list(_args) -> int:
for identity_file in sorted(BIN_DIR.glob("*/identity.json")):
identity = json.loads(identity_file.read_text())
print(f"{identity['name']:24} commit {identity['commit']:12} build {identity['build']:8} sha256 {identity['sha256'][:12]} {identity['version']}")
return 0
def load_binary(name: str) -> tuple[Path, dict]:
"""A registered binary must still be the bytes that were registered."""
identity_file = BIN_DIR / name / "identity.json"
if not identity_file.exists():
raise SystemExit(f"binary {name!r} is not registered; see `suite.py bin add`")
identity = json.loads(identity_file.read_text())
binary = BIN_DIR / name / "rustfs"
actual = h.sha256_hex(binary.read_bytes())
if actual != identity["sha256"]:
raise SystemExit(f"binary {name!r} changed since registration: sha256 {actual[:12]} != {identity['sha256'][:12]}")
return binary, identity
# ---------------------------------------------------------------------------
# Run
# ---------------------------------------------------------------------------
def _prune_runs() -> None:
"""Keep the most recent runs and every run that did not succeed."""
finished = sorted((d for d in RUNS_DIR.glob("*") if (d / "DONE").exists()), key=lambda d: d.name)
succeeded = [d for d in finished if (d / "DONE").read_text().strip() == "success"]
for old in succeeded[:-KEEP_RUNS]:
shutil.rmtree(old / "sites", ignore_errors=True)
def _provision(site: h.Site, config: str) -> None:
if config in fw.KMS_CONFIGS:
h.wait_kms_healthy(site)
for name in (h.KEY_1, h.KEY_2):
h.ensure_key(site, name)
def _run_case(case: fw.Case, site: h.Site, config: str, identity: dict) -> tuple[dict, list[dict], list[dict]]:
ctx = fw.Ctx(site=site, config=config, case_id=case.case_id, master_key_event=identity.get("master_key_event", True))
started = time.monotonic()
verdict, reasons = fw.PASS, []
observe.begin_case(config, case.case_id)
try:
if not h.is_running(site):
raise h.EnvError(f"{site.name} is not running")
case.fn(ctx)
reasons = list(ctx.failures)
except h.EnvError as err:
verdict, reasons = fw.ENV_ERROR, [str(err)]
except h.CheckFailed as err:
reasons = [*ctx.failures, str(err)]
except Exception: # noqa: BLE001 - an unexpected response shape must surface with its traceback, not vanish
reasons = [*ctx.failures, "unexpected exception:\n" + traceback.format_exc(limit=6)]
finally:
observe.end_case()
if verdict != fw.ENV_ERROR:
if reasons:
verdict = fw.FAIL
elif ctx.fixed:
verdict, reasons = fw.FIXED, [f"now holds: {text}" for text in ctx.fixed]
elif ctx.known:
verdict, reasons = fw.KNOWN, [f"does not hold: {text}" for text in ctx.known]
base = {"config": config, "case": case.case_id, "level": case.level, "layer": case.layer}
result = {**base, "row": "", "title": case.title, "verdict": verdict, "reasons": reasons,
"seconds": round(time.monotonic() - started, 2)}
rows = [] if verdict == fw.ENV_ERROR else [{**base, **row} for row in ctx.rows]
return result, rows, observe.drain()
def _build_self_check(identity: dict, observations: list[dict]) -> str | None:
"""A binary's stored form must match the build it was registered as; otherwise the run tested the wrong binary."""
stored = [o["value"] for o in observations if o["kind"] == "stored-keys" and o["value"]]
managed = [keys for keys in stored if any("encryption-key" in k or "sealed-key" in k for k in keys)]
if not managed:
return None
sealed = any("sealed-key" in k for keys in managed for k in keys)
declared_sealed = identity["build"] == "rio-v2"
if sealed != declared_sealed:
return (f"binary registered as build {identity['build']!r} but its objects "
f"{'carry' if sealed else 'do not carry'} sealed object-key metadata")
return None
def cmd_run(args) -> int:
binary, identity = load_binary(args.binary)
h.STATE_DIR.mkdir(parents=True, exist_ok=True)
lock = (h.STATE_DIR / "lock").open("w")
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise SystemExit("another run holds the lock; only one driver may run at a time")
selected = set(args.cases.split(",")) if args.cases else None
configs = [c for c in fw.CONFIG_ORDER if not args.configs or c in args.configs.split(",")]
run_as = dict(pair.split("=", 1) for pair in args.run_as.split(",")) if args.run_as else {}
for name in [*run_as, *run_as.values(), *(args.configs.split(",") if args.configs else [])]:
if name not in fw.CONFIGS:
raise SystemExit(f"unknown configuration {name!r}; known: {', '.join(fw.CONFIG_ORDER)}")
run_id = f"{datetime.datetime.now().strftime('%Y%m%dT%H%M%S')}-{identity['sha256'][:8]}"
run_dir = RUNS_DIR / run_id
run_dir.mkdir(parents=True)
_prune_runs()
meta = {"run_id": run_id, "level": args.level, "binary": identity, "port": args.port, "configs": configs,
"run_as": run_as, "sabotage": args.sabotage, "cases": sorted(selected) if selected else "all",
"started": datetime.datetime.now().isoformat(timespec="seconds"),
"harness": {p.name: h.sha256_hex(p.read_bytes())[:16] for p in sorted(h.HERE.glob("*.py"))}}
print(f"run {run_id}: binary {identity['name']} ({identity['version']}, build {identity['build']}), port {args.port}")
results, observations = [], []
for config in configs:
cases = [c for c in fw.CASES if config in c.configs and c.level == args.level
and (selected is None or c.case_id in selected)]
cases.sort(key=lambda c: c.last)
if not cases:
continue
site = h.Site(name=config, root=run_dir / "sites" / config, port=args.port)
atexit.register(h.stop_site, site)
env_config = run_as.get(config, config)
print(f"-- {config}" + (f" (started as {env_config})" if env_config != config else ""))
startup_error = None
try:
if env_config in fw.KMS_CONFIGS:
h.write_local_key_file(site.key_dir, h.DEFAULT_KEY)
h.start_site(site, binary, fw.CONFIGS[env_config](site))
_provision(site, env_config)
if args.sabotage == "kill-site":
h.stop_site(site)
except h.EnvError as err:
startup_error = str(err)
print(f" environment error: {startup_error.splitlines()[0]}")
for case in cases:
if startup_error is not None:
result = {"config": config, "case": case.case_id, "level": case.level, "layer": case.layer, "row": "",
"title": case.title, "verdict": fw.ENV_ERROR, "reasons": [startup_error], "seconds": 0.0}
rows, obs = [], []
else:
result, rows, obs = _run_case(case, site, config, identity)
results.extend([result, *rows])
observations.extend(obs)
print(f" {result['verdict']:9} {case.case_id:8} {case.title} ({result['seconds']}s)")
for reason in result["reasons"][:6]:
print(f" - {reason.splitlines()[0][:260]}")
if args.fail_fast and result["verdict"] not in (fw.PASS, fw.KNOWN):
break
h.stop_site(site)
self_check = _build_self_check(identity, observations)
case_results = [r for r in results if r["row"] == ""]
counts = {v: sum(1 for r in case_results if r["verdict"] == v)
for v in (fw.PASS, fw.FAIL, fw.ENV_ERROR, fw.SKIP, fw.KNOWN, fw.FIXED)}
status = "success"
if counts[fw.ENV_ERROR] or self_check:
status = "environment-error"
elif counts[fw.FAIL] or counts[fw.FIXED]:
# A deviation that no longer reproduces fails the run until its marker is turned into a plain check.
status = "failed"
elif not case_results:
status = "environment-error" # nothing ran: never a success
meta.update({"finished": datetime.datetime.now().isoformat(timespec="seconds"), "counts": counts, "status": status,
"build_self_check": self_check or "ok"})
(run_dir / "run.json").write_text(json.dumps(meta, indent=1) + "\n")
(run_dir / "results.jsonl").write_text("".join(json.dumps(r, sort_keys=True) + "\n" for r in results))
(run_dir / "observations.jsonl").write_text("".join(json.dumps(o, sort_keys=True) + "\n" for o in observations))
report = [f"# SSE black-box run {run_id}", "",
f"- binary: {identity['name']} ({identity['version']}), commit {identity['commit']}, build {identity['build']}, sha256 {identity['sha256'][:16]}",
f"- status: {status}; " + ", ".join(f"{k} {v}" for k, v in counts.items()),
f"- build self-check: {self_check or 'ok'}", "", "| config | case | verdict | seconds | reasons |", "|---|---|---|---|---|"]
for r in case_results:
reasons = "<br>".join(reason.splitlines()[0][:200].replace("|", "\\|") for reason in r["reasons"])
report.append(f"| {r['config']} | {r['case']} | {r['verdict']} | {r['seconds']} | {reasons} |")
(run_dir / "report.md").write_text("\n".join(report) + "\n")
(run_dir / "DONE").write_text(status + "\n")
print(f"{status}: " + ", ".join(f"{k} {v}" for k, v in counts.items()) + f", {len(observations)} observations")
if self_check:
print(f"build self-check failed: {self_check}")
print(f"run directory: {run_dir}")
return 0 if status == "success" else 1
# ---------------------------------------------------------------------------
# Cross: one binary writes, another reads the same data directory
# ---------------------------------------------------------------------------
def _cross_objects(config: str) -> list[tuple[str, dict | None, dict | None, dict, list[bytes]]]:
"""(key, write headers, read headers, projection, parts) for every stored form the configuration can produce."""
key_a = h.derived_key("ssec a")
schemes = [("sse-c", h.ssec(key_a), h.ssec(key_a), h.projection_c(key_a)), ("sse-s3", h.sse_s3(), None, h.projection_s3())]
if config == "e2":
schemes.append(("sse-kms", h.sse_kms(h.KEY_1), None, h.projection_kms(h.KEY_1)))
objects = [("plain", None, None, {}, [h.plain("cross plain", 70 * h.KIB)])]
for name, write_headers, read_headers, want in schemes:
# Sizes on both sides of the inline threshold and of the 1 MiB block boundary.
for size in (0, 1, 32 * h.KIB, 70 * h.KIB, 200_000, h.MIB, 3 * h.MIB + 1):
objects.append((f"{name}-{size}", write_headers, read_headers, want, [h.plain(f"cross {name} {size}", size)]))
parts = [h.plain(f"cross {name} part 1", 5 * h.MIB), h.plain(f"cross {name} part 2", h.MIB + 17)]
objects.append((f"{name}-multipart", write_headers, read_headers, want, parts))
return objects
def _cross_write(site: h.Site, bucket: str, objects) -> None:
"""Setup done by the writer binary; a failure here is not the reader's verdict."""
for key, write_headers, _, _, parts in objects:
part_headers = write_headers if write_headers and any(k.startswith(h.SSEC) for k in write_headers) else None
if len(parts) == 1:
status, _, body = h.put(site, bucket, key, parts[0], write_headers)
else:
status, _, upload_id, body = h.create_multipart(site, bucket, key, write_headers)
etags = []
for number, data in enumerate(parts, start=1):
if status != 200:
break
status, part_response, body = h.upload_part(site, bucket, key, upload_id, number, data, part_headers)
etags.append((number, part_response.get("etag", "")))
if status == 200:
status, _, body = h.complete_multipart(site, bucket, key, upload_id, etags, part_headers)
if status != 200:
raise h.EnvError(f"writer could not store {key}: HTTP {status} {h.s3_error(body)}")
def cmd_cross(args) -> int:
writer, writer_identity = load_binary(args.writer)
reader, reader_identity = load_binary(args.reader)
lock = (h.STATE_DIR / "lock").open("w")
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise SystemExit("another run holds the lock; only one driver may run at a time") from None
run_id = f"{datetime.datetime.now().strftime('%Y%m%dT%H%M%S')}-cross-{writer_identity['sha256'][:8]}-{reader_identity['sha256'][:8]}"
run_dir = RUNS_DIR / run_id
run_dir.mkdir(parents=True)
print(f"cross {run_id}: {writer_identity['name']} writes, {reader_identity['name']} reads")
results, observations = [], []
for config in ("e1", "e2"):
site = h.Site(name=config, root=run_dir / "sites" / config, port=args.port)
atexit.register(h.stop_site, site)
objects = _cross_objects(config)
bucket = f"bb-cross-{config}"
case = fw.Case(f"CROSS-{config}", f"{writer_identity['name']} writes, {reader_identity['name']} reads", "p1", "L1",
(config,), None)
def read_all(c: fw.Ctx, objects=objects, bucket=bucket) -> None:
for key, _, read_headers, want, parts in objects:
with c.row(key):
c.read_back(bucket, key, b"".join(parts), read_headers, want, key)
status, head_headers = h.head(c.site, bucket, key, read_headers)
if c.check(status == 200, f"{key} HEAD: HTTP {status}, expected 200"):
c.projection(head_headers, want, f"{key} HEAD")
c.stored(bucket, key)
case.fn = read_all
try:
if config in fw.KMS_CONFIGS:
h.write_local_key_file(site.key_dir, h.DEFAULT_KEY)
h.start_site(site, writer, fw.CONFIGS[config](site))
_provision(site, config)
h.make_bucket(site, bucket)
_cross_write(site, bucket, objects)
h.stop_site(site)
h.start_site(site, reader, fw.CONFIGS[config](site))
_provision(site, config)
result, rows, obs = _run_case(case, site, config, reader_identity)
except h.EnvError as err:
result = {"config": config, "case": case.case_id, "level": "p1", "layer": "L1", "row": "", "title": case.title,
"verdict": fw.ENV_ERROR, "reasons": [str(err)], "seconds": 0.0}
rows, obs = [], []
h.stop_site(site)
results.extend([result, *rows])
observations.extend(obs)
print(f" {result['verdict']:9} {case.case_id}: {sum(1 for r in rows if r['verdict'] == fw.PASS)}/{len(objects)} objects read back")
for reason in result["reasons"][:8]:
print(f" - {reason.splitlines()[0][:260]}")
verdicts = {r["verdict"] for r in results if r["row"] == ""}
status = "success" if verdicts == {fw.PASS} else ("environment-error" if fw.ENV_ERROR in verdicts else "failed")
meta = {"run_id": run_id, "kind": "cross", "binary": reader_identity, "writer": writer_identity, "status": status}
(run_dir / "run.json").write_text(json.dumps(meta, indent=1) + "\n")
(run_dir / "results.jsonl").write_text("".join(json.dumps(r, sort_keys=True) + "\n" for r in results))
(run_dir / "observations.jsonl").write_text("".join(json.dumps(o, sort_keys=True) + "\n" for o in observations))
(run_dir / "DONE").write_text(status + "\n")
print(f"{status}\nrun directory: {run_dir}")
return 0 if status == "success" else 1
# ---------------------------------------------------------------------------
# Diff
# ---------------------------------------------------------------------------
def _resolve_run(value: str) -> Path:
path = Path(value)
return path if path.is_dir() else RUNS_DIR / value
def cmd_selftest(_args) -> int:
failures = selftest.run()
for failure in failures:
print(f"selftest failed: {failure}")
print("selftest: " + ("ok" if not failures else f"{len(failures)} failed"))
return 1 if failures else 0
def cmd_diff(args) -> int:
run_a, run_b = diffing.load_run(_resolve_run(args.a)), diffing.load_run(_resolve_run(args.b))
outcome = diffing.compare(run_a, run_b)
text = diffing.render(run_a, run_b, outcome)
print(text, end="")
if args.out:
Path(args.out).write_text(text)
if outcome["compared"]["exchanges"] == 0:
print("nothing was compared: the two runs share no HTTP exchange")
return 2
return 1 if outcome["blocking"] else 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = parser.add_subparsers(dest="command", required=True)
bin_parser = sub.add_parser("bin", help="register or list server binaries")
bin_sub = bin_parser.add_subparsers(dest="bin_command", required=True)
add = bin_sub.add_parser("add")
add.add_argument("--name", required=True)
add.add_argument("--path", required=True)
add.add_argument("--commit", required=True)
add.add_argument("--build", required=True, choices=("default", "rio-v2"))
add.add_argument("--no-master-key-event", action="store_true",
help="the binary predates the sse_local_master_key startup event")
add.set_defaults(fn=cmd_bin_add)
bin_sub.add_parser("list").set_defaults(fn=cmd_bin_list)
run = sub.add_parser("run", help="run the cases against a registered binary")
run.add_argument("--binary", required=True)
run.add_argument("--level", default="p0", choices=("p0",))
run.add_argument("--configs", help="comma-separated subset of: " + ", ".join(fw.CONFIG_ORDER))
run.add_argument("--cases", help="comma-separated case ids")
run.add_argument("--port", type=int, default=h.DEFAULT_PORT)
run.add_argument("--fail-fast", action="store_true")
run.add_argument("--run-as", help="self-test: start configuration X with the environment of Y, as X=Y[,...]")
run.add_argument("--sabotage", choices=("kill-site",), help="self-test: break the environment on purpose")
run.set_defaults(fn=cmd_run)
cross = sub.add_parser("cross", help="one binary writes every stored form, another reads the same data directory")
cross.add_argument("--writer", required=True)
cross.add_argument("--reader", required=True)
cross.add_argument("--port", type=int, default=h.DEFAULT_PORT)
cross.set_defaults(fn=cmd_cross)
sub.add_parser("selftest", help="check offline that the suite tells apart what it must").set_defaults(fn=cmd_selftest)
diff = sub.add_parser("diff", help="compare two finished runs")
diff.add_argument("a")
diff.add_argument("b")
diff.add_argument("--out")
diff.set_defaults(fn=cmd_diff)
args = parser.parse_args()
return args.fn(args)
if __name__ == "__main__":
sys.exit(main())