gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-02 19:39:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: state that MinIO-encrypted objects do not migrate (#5600)

Browse Source 
* docs: state that MinIO-encrypted objects are not readable by RustFS

Operators evaluating a MinIO migration had no warning that objects MinIO
wrote with SSE-S3, SSE-KMS, or SSE-C cannot be read back. The container
formats interoperate, so the limitation is easy to discover only after
the data has moved.

Document the limitation where a migration decision is actually made:

- minio-file-format-compat.md gains Part C, covering which object classes
  transfer, the three seams that block each SSE mode with file:line
  evidence, the reverse direction, and the current workarounds. It also
  records that the `rio-v2` MinIO sealed-key parser does not close the
  gap: the feature is absent from released artifacts, and the managed-SSE
  detection gate is not feature-gated and returns before the parser runs.
- kms-backend-security.md gains an operator-facing warning next to the
  backend comparison table, since configuring the static backend with
  MinIO's key material looks like it should work and does not.
- s3-compatibility-matrix.md scopes its SSE row to RustFS's own
  round-trip.

The read path treats an undetected MinIO-encrypted object as unencrypted
rather than failing, so all three notes tell operators to verify migrated
objects by content instead of by status code.

Refs rustfs/backlog#1638.

* docs: correct the failure mode for MinIO-encrypted objects

The read path does fail closed; the earlier text claimed ciphertext was
served as plaintext. MinIO's internal headers mark the object encrypted,
so the reader refuses when no material resolves. What is actually wrong
is the diagnosis: the refusal surfaces as a 500 InternalError.
This commit is contained in:
Zhengchao An
2026-08-02 11:30:53 +08:00
committed by GitHub
parent 8bb147cb70
commit 34f1d2c0cd
3 changed files with 104 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/minio-file-format-compat.md
+79 -2
View File
@@ -26,9 +26,11 @@ Refs rustfs/backlog#580.
- **Migration**: RustFS already ships a one-way importer that reads a legacy
meta bucket and rewrites bucket-metadata + IAM config into the RustFS meta
bucket (`crates/ecstore/src/bucket/migration.rs`).
- **Server-side encryption**: not covered by the above. Objects MinIO wrote with SSE-S3, SSE-KMS, or SSE-C are **not readable by RustFS** in any shipped build. See [Part C](#part-c--server-side-encryption-sse) before planning a migration that includes encrypted objects.
The remaining work is verification breadth and closing per-config parsing gaps,
not a format rewrite.
For unencrypted objects the remaining work is verification breadth and closing
per-config parsing gaps, not a format rewrite. Encrypted objects are a separate,
unsolved axis (rustfs/backlog#1638).
---
@@ -220,6 +222,81 @@ missing piece is a source adapter that points the importer at a MinIO
---
## Part C — Server-Side Encryption (SSE)
Container-format parity does **not** extend to encrypted object payloads. RustFS currently does not support reading objects that MinIO wrote with server-side encryption — SSE-S3, SSE-KMS, or SSE-C. This is true of every released binary and container image. Tracked in rustfs/backlog#1638.
Note the asymmetry with Parts A and B: the `xl.meta` around a MinIO SSE object parses fine, so such objects list, HEAD, and report plausible sizes. Only the payload is unreadable.
### What can and cannot be migrated
| Object class | Readable after moving the drives / copying via S3 | Notes |
|---|:--:|---|
| Unencrypted objects | ✅ | Parts A and B apply. |
| Bucket metadata, IAM config | ✅ | Via the importer, once a `.minio.sys` source adapter exists (see Part B). |
| Bucket-level default-encryption *configuration* | ✅ | The `encryption` config blob round-trips as a blob; it does not make existing ciphertext readable. |
| MinIO-written SSE-S3 objects | ❌ | Seams 1 and 2 below. |
| MinIO-written SSE-KMS objects | ❌ | Seams 1 and 2 below. |
| MinIO-written SSE-C objects | ❌ | Seam 3 below. |
| RustFS-written SSE objects read back by MinIO | ❌ | See "Reverse direction". |
### Where the read path stops
The primitives match — RustFS implements the same DARE V2 stream format and the same object-key derivation and sealing, and a MinIO sealed-key parser exists (`parse_minio_managed_sealed_key`, `rustfs/src/storage/sse.rs:3195`). Three seams above the cryptography still reject MinIO-written objects.
| # | Seam | Evidence |
|---|---|---|
| 1 | The managed-SSE (SSE-S3 / SSE-KMS) read path returns "not encrypted" unless the object's *persisted* metadata carries the S3 response key `x-amz-server-side-encryption`. RustFS writes that key into metadata on PUT; MinIO's internal sealed-key headers alone do not satisfy the gate. | Gate: `rustfs/src/storage/sse.rs:2432`. RustFS write side: `rustfs/src/storage/sse.rs:391-400`. |
| 2 | MinIO's wrapped-DEK blob (`{"aead": ...}`) is neither produced nor accepted. `is_data_key_envelope` classifies that shape as not a RustFS envelope, and `LocalSseDekEnvelope` is `deny_unknown_fields`. | `crates/kms/src/encryption/dek.rs:425`, `:443`; `rustfs/src/storage/sse.rs:2784-2790`. Already documented for the static backend at `crates/kms/src/config.rs:304-308`. |
| 3 | SSE-C detection keys on `x-amz-server-side-encryption-customer-algorithm`, and `contains_managed_encryption_metadata` omits MinIO's SSE-C sealed-key header, so a MinIO SSE-C object matches neither detection branch. The unsealing code it would need is already written. | Detection: `rustfs/src/storage/sse.rs:2059` and `:3178-3184`; the omitted constant is `rustfs/src/storage/sse.rs:124`. Unsealing: `rustfs/src/storage/sse.rs:2236-2245`. |
### How it fails
The read fails closed: ciphertext is never served as plaintext. Seams 1 and 3 return `Ok(None)`, but that value does not reach the data path. `is_object_encryption_marker` matches the whole `x-minio-internal-server-side-encryption-` prefix (`crates/utils/src/http/header_compat.rs:50-67`), so `ObjectInfo::is_encrypted()` is true for these objects, and `crates/ecstore/src/object_api/readers.rs:559-568` turns the `Ok(None)` into `encrypted object metadata is incomplete` while constructing the reader. GET, CopyObject, replication and multipart sources all build the reader through that path. The inline fast path and the body cache both exclude encrypted objects explicitly, so neither bypasses it.
What migrates badly is the *diagnosis*, not the data. That error is not recognised by `map_get_object_reader_error` (`rustfs/src/storage/sse.rs:667`), so it surfaces as a 500 `InternalError` — which reads as a RustFS fault rather than "this object was encrypted by another implementation". List and HEAD still succeed, because `xl.meta` itself parses normally, so the object looks healthy until something reads it.
Seam 2 surfaces its own error, but only for objects that got past seam 1.
### The `rio-v2` feature does not change this
`rustfs/src/storage/sse.rs` contains MinIO-interop code behind `#[cfg(feature = "rio-v2")]`, which can give the impression that enabling the feature closes the gap. It does not, for two independent reasons.
- The feature is not compiled into anything that ships. `rio-v2` is absent from both `default` and `full` in `rustfs/Cargo.toml:39`, `:48`, `:51`; release binaries are built with no `--features` flag, and the published images install that binary rather than compiling their own.
- Seam 1 is not feature-gated and runs *before* the MinIO parser is consulted (`rustfs/src/storage/sse.rs:2432` precedes `:2458`). Even with `rio-v2` enabled, a MinIO-written managed-SSE object returns at the gate and never reaches `parse_minio_managed_sealed_key`.
The interop harness reflects this. The reader tests are `#[ignore]` (`rustfs/src/storage/minio_generated_read_test.rs:244`, `:250`), the workflow that would run them is disabled at the GitHub Actions level and states in its own header that end-to-end MinIO-to-RustFS SSE interop is not implemented (`.github/workflows/minio-interop.yml:24-29`, `:34-39`), and the fixture suite's scope note says the tests "do not yet validate full plaintext reconstruction from MinIO-written encrypted data" (`crates/rio-v2/tests/README.md:55`).
### Reverse direction
Migrating back is also unsupported. Under `rio-v2` RustFS writes its own DEK envelope into MinIO's sealed-key metadata slots and labels it with MinIO's seal algorithm (`rustfs/src/storage/sse.rs:1830-1852`), so the metadata is MinIO-shaped while the key bytes are not MinIO-openable. Default builds do not populate those slots at all (`rustfs/src/storage/sse.rs:1796-1798`). Treat RustFS-written SSE objects as readable only by RustFS.
### Working around the limitation
Until rustfs/backlog#1638 lands, the options are:
- Decrypt on the MinIO side first: rewrite the affected objects as plaintext (or copy them out through MinIO's S3 endpoint, which decrypts on read) and migrate the plaintext, applying RustFS-side encryption afterwards.
- Copy through the S3 API rather than moving drives: a client that reads from MinIO and writes to RustFS gets plaintext from the source and lets RustFS encrypt with its own KMS. This re-encrypts rather than preserving ciphertext, and costs a full data transfer.
- Leave encrypted objects on MinIO and migrate only unencrypted data.
Inventory the source first — bucket default-encryption settings mean objects can be encrypted without the uploader having asked for it, so "we never set SSE headers" is not sufficient evidence that a bucket has no encrypted objects.
### SSE interop verdict
| Item | Done | Partial | Todo |
|---|:--:|:--:|:--:|
| DARE V2 stream format parity | ✅ | | |
| Object-key derivation / sealing parity | ✅ | | |
| MinIO sealed-key parser exists (behind `rio-v2`) | | ⚠️ | |
| Managed-SSE detection accepts MinIO-written metadata | | | ❌ |
| MinIO `{"aead": ...}` wrapped-DEK parser | | | ❌ |
| SSE-C detection accepts MinIO-written metadata | | | ❌ |
| Read MinIO-written SSE-S3 / SSE-KMS / SSE-C objects end to end | | | ❌ |
| RustFS-written SSE objects readable by MinIO | | | ❌ |
| CI proof of SSE read parity | | | ❌ |
---
## Phased Plan
The format is already close; the plan is verification, a source adapter, and
docs/architecture/s3-compatibility-matrix.md
+2
View File
@@ -50,6 +50,8 @@ The implemented test list currently covers the common object-storage surface:
| SSE-C and selected SSE-KMS edge cases | Supported | `implemented_tests.txt` |
| Selected versioning, object-lock, checksum, CORS, raw request, and conditional write behavior | Supported | `implemented_tests.txt` |
"Supported" for the SSE row means RustFS encrypts and decrypts its own objects. It does not mean RustFS can read objects another implementation encrypted: objects MinIO wrote with SSE-S3, SSE-KMS, or SSE-C are not readable by RustFS today, which matters when migrating. See [MinIO file-format interoperability, Part C](minio-file-format-compat.md#part-c--server-side-encryption-sse) and rustfs/backlog#1638.
## Planned Standard Coverage
These are standard S3 areas that remain planned work and must not be described
docs/operations/kms-backend-security.md
+23 -1
View File
@@ -2,7 +2,7 @@
RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone.
For how the Vault backends authenticate (static token, AppRole, Vault Agent token file) and how credential refresh and the fail-closed window behave, see the [Vault KMS authentication runbook](vault-kms-authentication.md). For what may be claimed about the cryptographic implementations themselves, see [Cryptographic compliance positioning](kms-cryptographic-compliance.md). For which RustFS identities may manage or use a given key, see [Per-key KMS authorization](kms-per-key-authorization.md).
For how the Vault backends authenticate (static token, AppRole, Vault Agent token file) and how credential refresh and the fail-closed window behave, see the [Vault KMS authentication runbook](vault-kms-authentication.md). For what may be claimed about the cryptographic implementations themselves, see [Cryptographic compliance positioning](kms-cryptographic-compliance.md). For which RustFS identities may manage or use a given key, see [Per-key KMS authorization](kms-per-key-authorization.md). If you are migrating from MinIO, read [Migrating from MinIO: encrypted objects do not carry over](#migrating-from-minio-encrypted-objects-do-not-carry-over) first.
## Backend comparison
@@ -14,6 +14,28 @@ For how the Vault backends authenticate (static token, AppRole, Vault Agent toke
| Vault Transit | `VaultTransit` | Key-encryption keys never leave Vault; only Transit ciphertext is visible outside | Vault Transit engine (cryptographic isolation) | Delegated to Vault storage | Via Vault Transit key versioning | Deployments that need key material to be unreadable through storage APIs |
| AWS KMS | `AWS` (alias `AwsKms`) | Key material never leaves AWS KMS; RustFS mirrors no key state | AWS KMS (cryptographic isolation) + IAM | Delegated to AWS | On-demand `RotateKeyOnDemand`; prior backing keys stay usable for decryption | Deployments already rooted in AWS IAM that want AWS as the cryptographic root — read [AWS KMS: deviations from the shared backend contract](#aws-kms-deviations-from-the-shared-backend-contract) first |
## Migrating from MinIO: encrypted objects do not carry over
> **Warning: RustFS does not currently support reading objects that MinIO encrypted.**
> This applies to SSE-S3, SSE-KMS, and SSE-C, in every released binary and container image, and it holds regardless of which KMS backend you configure. Configuring the `Static` backend with the same key material MinIO used does **not** make those objects readable — MinIO wraps data keys in a different envelope format that no RustFS backend produces or accepts (`crates/kms/src/config.rs:304-308`). Plan for this **before** moving data. Tracked in rustfs/backlog#1638.
The read does fail closed — ciphertext is never served as plaintext. MinIO's internal encryption headers mark the object as encrypted (`crates/utils/src/http/header_compat.rs:50-67`), so the read path demands encryption material and refuses when none resolves (`crates/ecstore/src/object_api/readers.rs:559-568`). Two properties still make the problem easy to discover late:
- **The error does not say what happened.** It surfaces as a 500 `InternalError`, which reads as a RustFS fault rather than "another implementation encrypted this object".
- **Surrounding metadata migrates fine.** The object's `xl.meta` parses, so encrypted objects list and HEAD normally and report plausible sizes. The failure appears only when something reads the payload.
Read a sample of encrypted objects, not just their listings, before decommissioning the MinIO deployment.
Current options for a migration whose source contains encrypted objects:
- Decrypt on the MinIO side first, migrate plaintext, then let RustFS re-encrypt with its own KMS.
- Copy through the S3 API rather than moving drives — MinIO decrypts on read, and RustFS encrypts on write. This re-encrypts rather than preserving ciphertext and costs a full data transfer.
- Leave encrypted objects on MinIO and migrate only unencrypted data.
Inventory the source before choosing: bucket default-encryption settings mean objects can be encrypted without any client having sent SSE headers.
The same limitation applies in reverse — objects RustFS encrypts are not readable by MinIO. For the code-level breakdown of which seams block each SSE mode, see [MinIO file-format interoperability, Part C](../architecture/minio-file-format-compat.md#part-c--server-side-encryption-sse).
## Vault KV2: what the backend does and does not do
The Vault KV2 backend uses Vault purely as a **secure storage** service: