fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)

* Add black-box behavior tests for KMS resilience and serialization * fix(kms): repair unopenable ciphertext across backends Black-box testing of the KMS crate surfaced several defects that make encrypted data permanently unreadable. Symmetric envelopes. The Local and Vault Transit backends returned raw cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so anything sealed through the master-key path could never be opened again. Local also discarded the AES-GCM nonce. Both now emit the same envelope `decrypt` consumes, matching the Static backend. Deterministic AAD. The object layer derived AEAD additional data by serializing a `HashMap` directly. Iteration order differs per instance, so a context rebuilt from storage produced different AAD bytes than the one used to seal and the object stopped opening. Ordering by key removes that dependency, matching the Static backend's existing `context_aad`. Objects written with the default single-key context are unaffected, since a one-entry map has only one serialization. Cipher in the header projection. `metadata_to_headers` recorded the SSE mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305, so a ChaCha-sealed object came back claiming `aws:kms` and was opened with the wrong cipher. The cipher now travels in `x-rustfs-encryption-algorithm` — the header the storage layer already reads but nothing ever wrote. Objects without it fall back as before. Also: the Static backend ignored `key_spec` and always issued 256-bit data keys; Local `list_keys` hardcoded `truncated: false`, ignored `marker`, and paginated over unordered `read_dir`, so a paginating client silently saw a partial key list; and Local and Vault KV2 reported `key_id: "unknown"` from `decrypt` despite the envelope naming the master key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(kms): cover both Vault backends and key rotation The behavior suite ran only against Local and Static, and its own harness documented the gap: the Vault backends had no business-capability coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and Vault Transit to every `for_each_backend` spec against a live server. That lane is what surfaced the Transit envelope defect fixed in the previous commit. `rotate` and `versioning` are advertised only by the Vault backends, so until now every capability-gated branch for them took the `UnsupportedCapability` side and the working half was never asserted — a rotation that dropped prior key versions would have gone green. The new `behavior_rotation.rs` pins that half: material sealed before a rotation still opens after it, repeated rotations accumulate versions rather than overwriting a single spare, and the history survives a restart. Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms` asserted a 1-byte object differs from its own ciphertext, which collides once every 256 runs; the assertion now applies only where a collision is not realistic, and small objects stay covered by the tag check and the decrypt round-trip. `test_from_env_selects_token_file` depended on `RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and now clears it explicitly. The snapshots directory was also removed from `.gitignore`: insta snapshots are the assertions themselves, so leaving them untracked gives CI nothing to compare against. Only `.snap.new` scratch files are ignored now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(kms): adapt behavior suite to current key APIs Rebasing onto main brought four API changes the suite predates. `DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now gated on the server's `allow_immediate_deletion`. Scheduled deletions pass `None`; the four specs that destroy a key outright echo the key id back and opt the harness config in, which is what the gate asks of a real caller. `LocalBackupExportRequest` gained `sanitized_config`. These specs cover the key-material path, so they seal no configuration and pass `None`. `KmsCacheStats` became a named struct with real hit, miss, and eviction counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data` existed to pin the old placeholder behavior — that the second tuple element was always zero — which main has since fixed, so it is now `cache_stats_reports_hits_and_misses_separately` and asserts the counters actually move. Starting the service provisions the reserved probe key, so it shows up in listings and backup bundles. Exact-set assertions filter it through a new `without_probe_key` helper rather than naming it, keeping those specs about the keys they seeded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(kms): bind the AAD to the stored context bytes Review caught that canonicalizing the AAD on decrypt breaks objects sealed before canonicalization existed, and it was right. The AAD is the *serialization* of the encryption context, and `x-rustfs-encryption-context` stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the AEAD and then moved the same map into the metadata the header is written from, so the stored string is byte-identical to the AAD the object was sealed under. Those objects are therefore recoverable — but only while nothing round-trips the value through a `HashMap` and re-serializes it. Recomputing sorted AAD on decrypt would have turned a readable object into a permanently unreadable one. The previous behavior was worse than the first analysis credited: it did not merely fail intermittently, it made the failure deterministic. `EncryptionMetadata` now carries `context_aad`, the bytes the object was actually sealed with. Encryption records what it fed the AEAD, the header projection stores those bytes verbatim (and preserves a legacy ordering across a re-projection rather than rewriting it into sorted form), and `headers_to_metadata` carries the stored string through untouched. Both decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical serialization only when no stored serialization exists. Canonicalization still applies to everything newly sealed, so the original ordering bug cannot recur. Two tests pin this: a legacy record whose sealed bytes are non-canonical must survive a full header round trip unchanged, and a context header rewritten to an equivalent-but-reordered serialization must fail authentication rather than silently re-deriving a working AAD. Both were mutation-checked against the reinstated bug on each side. Also from review: the lifecycle churn test asserted only that every request was accounted for, which holds whether the state gate exists or not, so both branches are now pinned deterministically after the churn (asserting `refused > 0` on the concurrent phase would only trade the hole for a scheduling flake). And the Local and Vault KV2 envelopes compare `encryption_context` without authenticating it — `DekCrypto` seals only the plaintext — which is now documented at both sites; closing it needs a versioned envelope, since existing ciphertext was sealed without AAD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 15:37:02 +00:00 · 2026-08-03 23:33:08 +08:00
parent d8d22599fe
commit 62cc19e937
36 changed files with 6292 additions and 32 deletions
@@ -1450,7 +1450,23 @@ impl LocalKmsClient {
        let key_info = self.describe_key(&request.key_id, context).await?;
        ensure_key_status_permits(&request.key_id, &key_info.status, StateGatedOperation::Encrypt)?;

-        let (ciphertext, _nonce) = self.encrypt_with_master_key(&request.key_id, &request.plaintext).await?;
+        let (encrypted_key, nonce) = self.encrypt_with_master_key(&request.key_id, &request.plaintext).await?;
+
+        // The ciphertext must be the same envelope `decrypt` parses: the nonce
+        // and the bound context live in it, so handing back the bare AES-GCM
+        // output would make every `encrypt` result permanently unopenable.
+        let envelope = DataKeyEnvelope {
+            key_id: uuid::Uuid::new_v4().to_string(),
+            master_key_id: request.key_id.clone(),
+            key_spec: key_info.algorithm.clone(),
+            encrypted_key,
+            nonce,
+            encryption_context: request.encryption_context.clone(),
+            created_at: Zoned::now(),
+            // Local rotation is rejected, so the key has a single material version.
+            master_key_version: None,
+        };
+        let ciphertext = serde_json::to_vec(&envelope)?;

        Ok(EncryptResponse {
            ciphertext,
@@ -1466,6 +1482,13 @@ impl LocalKmsClient {
        // Parse the data key envelope from ciphertext
        let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)?;

+        // NOTE: this comparison is an authorization check, not a cryptographic
+        // binding. `DekCrypto` seals only the plaintext, so `encryption_context`
+        // rides in the envelope unauthenticated: anyone able to rewrite the
+        // stored envelope can rewrite this field and present a matching context.
+        // The Static and Vault Transit backends do bind it (as AEAD AAD and as
+        // the Transit KDF context respectively); closing the gap here needs a
+        // versioned envelope, since existing ciphertext was sealed without AAD.
        // Verify encryption context matches
        // Check that all keys in envelope.encryption_context are present in request.encryption_context
        // and their values match. This ensures the context used for decryption matches what was used for encryption.
@@ -1849,10 +1872,14 @@ impl KmsBackend for LocalKmsBackend {
    async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse> {
        let plaintext = self.client.decrypt(&request, None).await?;

-        // For simplicity, return basic response - in real implementation would extract more info from ciphertext
+        // The envelope that was just opened names the master key that opened it.
+        // Reporting "unknown" left every caller unable to tell which key was
+        // actually used, which is what audit and key-rotation checks read.
+        let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)?;
+
        Ok(DecryptResponse {
            plaintext,
-            key_id: "unknown".to_string(), // Would be extracted from ciphertext metadata
+            key_id: envelope.master_key_id,
            encryption_algorithm: Some("AES-256-GCM".to_string()),
        })
    }
@@ -26,7 +26,7 @@

 use crate::backends::{BackendCapabilities, KmsBackend, empty_key_page, list_keys_page_size};
 use crate::config::{BackendConfig, KmsConfig};
-use crate::encryption::DataKeyEnvelope;
+use crate::encryption::{DataKeyEnvelope, context_aad};
 use crate::error::{KmsError, Result};
 use crate::types::*;
 use aes_gcm::{
@@ -36,7 +36,7 @@ use aes_gcm::{
 use async_trait::async_trait;
 use jiff::Zoned;
 use rand::RngExt;
-use std::collections::{BTreeMap, HashMap};
+use std::collections::HashMap;
 use tracing::debug;
 use zeroize::Zeroizing;

@@ -45,11 +45,6 @@ const NONCE_SIZE: usize = 12;
 /// AES-256 key size in bytes.
 const KEY_SIZE: usize = 32;

-fn context_aad(context: &HashMap<String, String>) -> Result<Vec<u8>> {
-    let canonical: BTreeMap<&str, &str> = context.iter().map(|(key, value)| (key.as_str(), value.as_str())).collect();
-    serde_json::to_vec(&canonical).map_err(Into::into)
-}
-
 /// Static single-key KMS backend.
 ///
 /// Uses a pre-configured AES-256 key to derive data encryption keys. This is a
@@ -113,8 +108,18 @@ impl StaticKmsBackend {
        let mut nonce_bytes = [0u8; NONCE_SIZE];
        rand::rng().fill(&mut nonce_bytes[..]);

-        // Generate 32 random bytes as plaintext DEK
-        let mut plaintext = [0u8; KEY_SIZE];
+        // The requested spec decides the DEK length; a caller that asked for
+        // AES_128 and silently got 256 bits would build objects whose recorded
+        // spec does not match their key material.
+        // Lengths track `KeySpec::key_size`; the request carries the spec as a
+        // string, so the mapping is repeated here rather than shared.
+        let key_length = match request.key_spec.as_str() {
+            "AES_256" | "ChaCha20" => 32,
+            "AES_128" => 16,
+            _ => return Err(KmsError::unsupported_algorithm(&request.key_spec)),
+        };
+
+        let mut plaintext = vec![0u8; key_length];
        rand::rng().fill(&mut plaintext[..]);

        // Encrypt DEK with AES-256-GCM using the static key directly
@@ -127,7 +132,7 @@ impl StaticKmsBackend {
            .encrypt(
                &nonce,
                Payload {
-                    msg: plaintext.as_ref(),
+                    msg: plaintext.as_slice(),
                    aad: &aad,
                },
            )
@@ -857,6 +857,13 @@ impl VaultKmsClient {
        let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)
            .map_err(|e| KmsError::cryptographic_error("parse", format!("Failed to parse data key envelope: {e}")))?;

+        // NOTE: this comparison is an authorization check, not a cryptographic
+        // binding. `DekCrypto` seals only the plaintext, so `encryption_context`
+        // rides in the envelope unauthenticated: anyone able to rewrite the
+        // stored envelope can rewrite this field and present a matching context.
+        // The Static and Vault Transit backends do bind it (as AEAD AAD and as
+        // the Transit KDF context respectively); closing the gap here needs a
+        // versioned envelope, since existing ciphertext was sealed without AAD.
        // Verify encryption context matches
        // Check that all keys in envelope.encryption_context are present in request.encryption_context
        // and their values match. This ensures the context used for decryption matches what was used for encryption.
@@ -1580,9 +1587,14 @@ impl KmsBackend for VaultKmsBackend {
    async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse> {
        let plaintext = self.client.decrypt(&request, None).await?;

+        // The envelope that was just opened names the master key that opened it.
+        // Reporting "unknown" left every caller unable to tell which key was
+        // actually used, which is what audit and key-rotation checks read.
+        let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)?;
+
        Ok(DecryptResponse {
            plaintext,
-            key_id: "unknown".to_string(), // Would be extracted from ciphertext metadata
+            key_id: envelope.master_key_id,
            encryption_algorithm: Some("AES-256-GCM".to_string()),
        })
    }
@@ -772,7 +772,7 @@ impl VaultTransitKmsClient {
        let metadata = self
            .ensure_key_state_allows(&request.key_id, StateGatedOperation::Encrypt)
            .await?;
-        let ciphertext = match self
+        let encrypted = match self
            .transit_encrypt(&request.key_id, &request.plaintext, &request.encryption_context)
            .await
        {
@@ -783,8 +783,25 @@ impl VaultTransitKmsClient {
            }
        };

+        // The ciphertext must be the same envelope `decrypt` parses — it is what
+        // carries the key id and the bound context. Returning the bare Transit
+        // string made every `encrypt` result permanently unopenable.
+        let envelope = DataKeyEnvelope {
+            key_id: uuid::Uuid::new_v4().to_string(),
+            master_key_id: request.key_id.clone(),
+            key_spec: "AES_256".to_string(),
+            encrypted_key: encrypted.into_bytes(),
+            nonce: Vec::new(),
+            encryption_context: request.encryption_context.clone(),
+            created_at: Zoned::now(),
+            // Transit ciphertext already self-describes its key version
+            // ("vault:vN:..."), so the envelope never carries one.
+            master_key_version: None,
+        };
+        let ciphertext = serde_json::to_vec(&envelope)?;
+
        Ok(EncryptResponse {
-            ciphertext: ciphertext.into_bytes(),
+            ciphertext,
            key_id: request.key_id.clone(),
            key_version: metadata.current_version,
            algorithm: "vault-transit".to_string(),
@@ -1698,7 +1715,9 @@ mod tests {
            )
            .await
            .expect("encrypt must retry past a transient 429");
-        assert_eq!(response.ciphertext, b"vault:v1:scripted".to_vec());
+        let envelope: DataKeyEnvelope = serde_json::from_slice(&response.ciphertext).expect("encrypt must return an envelope");
+        assert_eq!(envelope.encrypted_key, b"vault:v1:scripted".to_vec());
+        assert_eq!(envelope.master_key_id, "wired-key");

        let requests = vault.requests();
        assert_eq!(requests.len(), 3, "metadata read plus two encrypt attempts: {requests:?}");
@@ -1833,6 +1833,10 @@ mod tests {
                ("RUSTFS_KMS_BACKEND", Some("vault")),
                ("RUSTFS_KMS_VAULT_ADDRESS", Some("https://vault.example.com")),
                (ENV_KMS_VAULT_TOKEN_FILE, Some("/run/vault-agent/token")),
+                // Cleared explicitly: a static token in the ambient environment
+                // outranks the token file, so leaving it up to the caller's shell
+                // would make this assertion depend on who runs the test.
+                ("RUSTFS_KMS_VAULT_TOKEN", None),
            ],
            || {
                let config = KmsConfig::from_env().expect("kms config should load from env");
@@ -27,7 +27,7 @@ use jiff::Zoned;
 use rand::Rng;
 use serde::de::{self, IgnoredAny, MapAccess, Visitor};
 use serde::{Deserialize, Serialize};
-use std::collections::HashMap;
+use std::collections::{BTreeMap, HashMap};
 use std::fmt;
 use std::sync::atomic::{AtomicU64, Ordering};

@@ -229,6 +229,23 @@ struct DataKeyEnvelopeMarker {
    _created_at: IgnoredAny,
 }

+/// Serialize an encryption context into deterministic AAD bytes.
+///
+/// The AAD has to be reproducible byte-for-byte at decrypt time. A `HashMap`
+/// serializes in its own iteration order, which differs between instances — so
+/// a context rebuilt from storage (or from headers) would produce different
+/// bytes than the one used to seal, and the sealed data would never open
+/// again. Ordering by key removes that dependency.
+///
+/// Shared by every layer that binds a context as additional data. It lives
+/// here rather than beside one caller because a second, subtly different copy
+/// is exactly how the object layer ended up serializing a `HashMap` directly
+/// while the Static backend was already canonicalizing.
+pub fn context_aad(context: &HashMap<String, String>) -> Result<Vec<u8>> {
+    let canonical: BTreeMap<&str, &str> = context.iter().map(|(key, value)| (key.as_str(), value.as_str())).collect();
+    serde_json::to_vec(&canonical).map_err(Into::into)
+}
+
 /// Returns whether ciphertext is a RustFS KMS data-key envelope.
 pub fn is_data_key_envelope(ciphertext: &[u8]) -> bool {
    ciphertext.iter().copied().find(|byte| !byte.is_ascii_whitespace()) == Some(b'{')
@@ -17,4 +17,4 @@
 pub mod ciphers;
 pub mod dek;

-pub use dek::{AesDekCrypto, DataKeyEnvelope, DekCrypto, generate_key_material, is_data_key_envelope};
+pub use dek::{AesDekCrypto, DataKeyEnvelope, DekCrypto, context_aad, generate_key_material, is_data_key_envelope};
@@ -19,6 +19,7 @@ use crate::api_types::{
 };
 use crate::cache::KmsCacheStats;
 use crate::encryption::ciphers::{create_cipher, generate_iv};
+use crate::encryption::context_aad;
 use crate::error::{KmsError, Result};
 use crate::manager::KmsManager;
 use crate::types::*;
@@ -82,6 +83,15 @@ fn request_encryption_context(context: &ObjectEncryptionContext) -> HashMap<Stri

 const INTERNAL_ENCRYPTION_KEY_ID_HEADER: &str = "x-rustfs-encryption-key-id";

+/// Carries the AEAD algorithm the object was sealed with.
+///
+/// The S3 `x-amz-server-side-encryption` header records the *SSE mode*
+/// (`AES256` / `aws:kms`), not the cipher, so it cannot round-trip
+/// `ChaCha20Poly1305`. Without this header a ChaCha-sealed object comes back
+/// from the projection claiming `aws:kms` and is then opened with the wrong
+/// cipher.
+const INTERNAL_ENCRYPTION_ALGORITHM_HEADER: &str = "x-rustfs-encryption-algorithm";
+
 /// Result of object encryption
 #[derive(Debug, Clone)]
 pub struct EncryptionResult {
@@ -484,7 +494,7 @@ impl ObjectEncryptionService {
        let iv = generate_iv(algorithm);

        // Build AAD from encryption context
-        let aad = serde_json::to_vec(&context)?;
+        let aad = context_aad(&context)?;

        // Encrypt the data
        let (ciphertext, tag) = cipher.encrypt(&data, &iv, &aad)?;
@@ -498,6 +508,9 @@ impl ObjectEncryptionService {
            tag: Some(tag),
            encryption_context: context,
            encrypted_at: Zoned::now(),
+            // Pinned to the bytes actually fed to the AEAD, so the projection
+            // below can store them verbatim instead of re-deriving them.
+            context_aad: Some(aad),
            original_size,
            encrypted_data_key: data_key.ciphertext_blob,
        };
@@ -558,7 +571,13 @@ impl ObjectEncryptionService {
        let cipher = create_cipher(&algorithm, &decrypt_response.plaintext)?;

        // Build AAD from encryption context
-        let aad = serde_json::to_vec(&metadata.encryption_context)?;
+        // Prefer the bytes the object was sealed under. Deriving them from the
+        // parsed map would re-order a pre-canonicalization context and fail the
+        // AEAD on an object that is otherwise perfectly readable.
+        let aad = match metadata.context_aad.as_ref() {
+            Some(stored) => stored.clone(),
+            None => context_aad(&metadata.encryption_context)?,
+        };

        // Get tag from metadata
        let tag = metadata
@@ -634,7 +653,7 @@ impl ObjectEncryptionService {
            ("sse_type".to_string(), "customer".to_string()),
        ]);

-        let aad = serde_json::to_vec(&context)?;
+        let aad = context_aad(&context)?;

        // Encrypt the data
        let (ciphertext, tag) = cipher.encrypt(&data, &iv, &aad)?;
@@ -648,6 +667,9 @@ impl ObjectEncryptionService {
            tag: Some(tag),
            encryption_context: context,
            encrypted_at: Zoned::now(),
+            // Pinned to the bytes actually fed to the AEAD, so the projection
+            // below can store them verbatim instead of re-deriving them.
+            context_aad: Some(aad),
            original_size,
            encrypted_data_key: Vec::new(), // Empty for SSE-C
        };
@@ -702,7 +724,13 @@ impl ObjectEncryptionService {
        let cipher = create_cipher(&algorithm, customer_key)?;

        // Build AAD from encryption context
-        let aad = serde_json::to_vec(&metadata.encryption_context)?;
+        // Prefer the bytes the object was sealed under. Deriving them from the
+        // parsed map would re-order a pre-canonicalization context and fail the
+        // AEAD on an object that is otherwise perfectly readable.
+        let aad = match metadata.context_aad.as_ref() {
+            Some(stored) => stored.clone(),
+            None => context_aad(&metadata.encryption_context)?,
+        };

        // Get tag from metadata
        let tag = metadata
@@ -774,6 +802,9 @@ impl ObjectEncryptionService {
            headers.insert(INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), metadata.key_id.clone());
        }

+        // Record the cipher separately from the SSE mode advertised above.
+        headers.insert(INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), metadata.algorithm.clone());
+
        // Internal headers for decryption
        headers.insert(
            "x-rustfs-encryption-iv".to_string(),
@@ -792,9 +823,16 @@ impl ObjectEncryptionService {
            base64::engine::general_purpose::STANDARD.encode(&metadata.encrypted_data_key),
        );

+        // Whatever the object was sealed under is what gets stored: for a
+        // pre-canonicalization object that is its original ordering, which must
+        // survive a re-projection rather than being rewritten into sorted form.
+        let context_bytes = match metadata.context_aad.as_ref() {
+            Some(stored) => stored.clone(),
+            None => context_aad(&metadata.encryption_context).unwrap_or_default(),
+        };
        headers.insert(
            "x-rustfs-encryption-context".to_string(),
-            serde_json::to_string(&metadata.encryption_context).unwrap_or_default(),
+            String::from_utf8_lossy(&context_bytes).into_owned(),
        );

        headers
@@ -809,18 +847,27 @@ impl ObjectEncryptionService {
    /// EncryptionMetadata parsed from headers
    ///
    pub fn headers_to_metadata(&self, headers: &HashMap<String, String>) -> Result<EncryptionMetadata> {
-        let algorithm = headers
+        let sse_mode = headers
            .get("x-amz-server-side-encryption")
            .ok_or_else(|| KmsError::validation_error("Missing encryption algorithm header"))?
            .clone();

-        let key_id = if algorithm == "AES256" && headers.contains_key("x-amz-server-side-encryption-customer-algorithm") {
+        // Prefer the recorded cipher; fall back to the SSE mode for objects
+        // written before that header existed, where `AES256`/`aws:kms` was the
+        // only thing stored and AES-256-GCM was the only cipher in use.
+        let algorithm = match headers.get(INTERNAL_ENCRYPTION_ALGORITHM_HEADER) {
+            Some(algorithm) => algorithm.clone(),
+            None if sse_mode == "aws:kms" => EncryptionAlgorithm::Aes256.as_str().to_string(),
+            None => sse_mode.clone(),
+        };
+
+        let key_id = if sse_mode == "AES256" && headers.contains_key("x-amz-server-side-encryption-customer-algorithm") {
            "sse-c".to_string()
        } else if let Some(key_id) = headers.get(INTERNAL_ENCRYPTION_KEY_ID_HEADER) {
            key_id.clone()
        } else if let Some(kms_key_id) = headers.get("x-amz-server-side-encryption-aws-kms-key-id") {
            kms_key_id.clone()
-        } else if algorithm == "AES256" {
+        } else if sse_mode == "AES256" {
            self.get_default_key_id()
                .cloned()
                .ok_or_else(|| KmsError::validation_error("Missing key ID"))?
@@ -853,11 +900,17 @@ impl ObjectEncryptionService {
            Vec::new() // Empty for SSE-C
        };

-        let encryption_context = if let Some(context_str) = headers.get("x-rustfs-encryption-context") {
-            serde_json::from_str(context_str)
-                .map_err(|e| KmsError::validation_error(format!("Invalid encryption context: {e}")))?
-        } else {
-            HashMap::new()
+        // The stored string is the AAD verbatim. It is parsed into a map for
+        // callers that inspect the context, but the bytes are carried through
+        // untouched: re-serializing the parsed map is exactly how the original
+        // ordering — and with it the ability to open the object — was lost.
+        let (encryption_context, context_aad) = match headers.get("x-rustfs-encryption-context") {
+            Some(context_str) => (
+                serde_json::from_str(context_str)
+                    .map_err(|e| KmsError::validation_error(format!("Invalid encryption context: {e}")))?,
+                Some(context_str.as_bytes().to_vec()),
+            ),
+            None => (HashMap::new(), None),
        };

        Ok(EncryptionMetadata {
@@ -870,6 +923,7 @@ impl ObjectEncryptionService {
            encrypted_at: Zoned::now(),
            original_size: 0, // Not available from headers
            encrypted_data_key,
+            context_aad,
        })
    }
 }
@@ -986,6 +1040,8 @@ mod tests {
            encrypted_at: Zoned::now(),
            original_size: 100,
            encrypted_data_key: vec![1, 2, 3, 4],
+            // A hand-built record with no sealed bytes to defer to.
+            context_aad: None,
        };

        // Convert to headers
@@ -636,6 +636,20 @@ pub struct EncryptionMetadata {
    pub original_size: u64,
    /// Encrypted data key
    pub encrypted_data_key: Vec<u8>,
+    /// The exact AAD bytes this object was sealed under.
+    ///
+    /// The AAD is the serialized encryption context, and the serialization is
+    /// what must be reproduced byte-for-byte — not the map. Objects written
+    /// before the context was canonicalized carry whichever `HashMap` order
+    /// happened to be in effect when they were sealed, and
+    /// `x-rustfs-encryption-context` preserves that exact byte sequence. It is
+    /// therefore recoverable, but only while it is never round-tripped through
+    /// a `HashMap` and re-serialized.
+    ///
+    /// `None` means "derive it from `encryption_context`", which is correct
+    /// only when no stored serialization exists to defer to.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub context_aad: Option<Vec<u8>>,
 }

 /// Health status information