mirror of https://github.com/rustfs/rustfs.git synced 2026-09-21 01:53:30 +00:00

Files

T

唐小鸭 e2e6a2535a fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

* fix(sse): classify bare SSE-KMS writes when no KMS is available

A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.

Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.

Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.

(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)

* fix(sse): keep KMS error classes on the encrypted read path

GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.

Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.

Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.

(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)

* fix(kms): classify key-management errors on the admin routes

`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.

Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.

The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.

(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)

* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key

`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.

Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.

Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.

(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)

* fix(s3): validate PutBucketEncryption rules before storing them

A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.

Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.

Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.

(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)

* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes

* test: align two unit tests with the new KMS and bucket-encryption contracts

`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.

2026-09-12 23:48:10 +08:00

bucket_default_encryption_test.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

common.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

configured_roundtrip_test.rs

fix(kms): restore persisted configuration after restart (#6821 )

2026-08-29 16:21:22 +08:00

copy_object_self_copy_sse_test.rs

test(kms): replace 33 hard-coded startup sleeps with readiness probe (#6349 )

2026-08-22 16:35:57 +00:00

copy_object_version_restore_sse_test.rs

test(kms): replace 33 hard-coded startup sleeps with readiness probe (#6349 )

2026-08-22 16:35:57 +00:00

encrypted_range_get_test.rs

feat(ecstore): closed-form range seek for single-part v2 encrypted objects (#6601 )

2026-08-26 09:43:34 +08:00

encryption_metadata_test.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

kms_anonymous_enforcement_test.rs

fix(sse): document and lock anonymous denial under KMS key policy (#6739 )

2026-08-27 18:34:34 +08:00

kms_authorization_negative_matrix_test.rs

feat(ecstore): observe PUT commit lock admission (#6319 )

2026-08-21 01:18:53 +00:00

kms_comprehensive_test.rs

chore(deps): migrate direct encoding deps to simd (#6690 )

2026-08-27 03:17:24 +08:00

kms_edge_cases_test.rs

chore(deps): migrate direct encoding deps to simd (#6690 )

2026-08-27 03:17:24 +08:00

kms_fault_recovery_test.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

kms_ilm_sse_kms_test.rs

test(e2e): restore tier and inline full-suite checks (#6794 )

2026-08-28 21:08:30 +08:00

kms_local_test.rs

chore(deps): migrate direct encoding deps to simd (#6690 )

2026-08-27 03:17:24 +08:00

kms_rekey_sweep_test.rs

feat(kms): bulk DEK rekey sweep with admin API and kms:Rekey action (#6654 )

2026-08-26 18:19:28 +08:00

kms_service_stop_test.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

kms_vault_test.rs

chore(deps): migrate direct encoding deps to simd (#6690 )

2026-08-27 03:17:24 +08:00

mod.rs

fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697 )

2026-09-12 23:48:10 +08:00

multipart_encryption_test.rs

fix(s3): align encrypted checksums and multipart completion (#7025 )

2026-09-01 17:35:49 +00:00

README.md

fix(e2e): require a verified server binary for every e2e run (#7687 )

2026-09-12 11:23:06 +08:00

select_sse_response_test.rs

fix(s3select): return encryption response headers (#6819 )

2026-08-29 19:46:34 +08:00

README.md

KMS End-to-End Tests

This directory contains the integration suites used to validate the full RustFS KMS (Key Management Service) workflow.

📁 Test Overview

`kms_local_test.rs`

End-to-end coverage for the local KMS backend:

Auto-start and configure the local backend
Configure KMS through the dynamic configuration API
Verify SSE-C (client-provided keys)
Exercise S3-compatible encryption/decryption
Validate key lifecycle management

`kms_vault_test.rs`

End-to-end coverage for the Vault backend:

Launch a Vault dev server automatically
Configure the transit engine and encryption keys
Configure KMS via the dynamic configuration API
Run the full Vault integration flow
Validate token authentication and encryption operations

`kms_comprehensive_test.rs`

Full KMS capability suite (currently disabled because of AWS SDK compatibility issues):

Bucket encryption configuration: SSE-S3 and SSE-KMS defaults
All SSE encryption modes:
- SSE-S3 (S3-managed server-side encryption)
- SSE-KMS (KMS-managed server-side encryption)
- SSE-C (client-provided keys)
Object operations: upload, download, and validation for every SSE mode
Multipart uploads: cover each SSE mode
Object replication: cross-mode replication scenarios
Complete KMS API management:
- Key lifecycle (create, list, describe, delete, cancel delete)
- Direct encrypt/decrypt operations
- Data key generation and handling
- KMS service lifecycle (start, stop, status)

`kms_integration_test.rs`

Broad integration tests that exercise:

Multiple backends
KMS lifecycle management
Error handling and recovery
Note: currently disabled because of AWS SDK compatibility gaps

🚀 Running Tests

Prerequisites

System dependencies

# macOS
brew install vault awscurl

# Ubuntu/Debian
apt-get install vault
pip install awscurl

Build RustFS (from the repository root)
```
python3 scripts/e2e_binary.py build
```

Run individual suites

Run every command below from the repository root through scripts/e2e_binary.py run; plain cargo test -p e2e_test fails with a missing E2E run receipt.

Local backend

python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_local_kms_end_to_end -- --nocapture

Vault backend

python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_vault_kms_end_to_end -- --nocapture

High availability

python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_vault_kms_high_availability -- --nocapture

Comprehensive features (disabled)

# Disabled due to AWS SDK compatibility gaps
# python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_comprehensive_kms_functionality -- --nocapture
# python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_sse_modes_compatibility -- --nocapture
# python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_kms_api_comprehensive -- --nocapture

Run all KMS suites

python3 scripts/e2e_binary.py run -- cargo test -p e2e_test kms -- --nocapture

Run serially (avoid port conflicts)

python3 scripts/e2e_binary.py run -- cargo test -p e2e_test kms -- --nocapture --test-threads=1

🔧 Configuration

Environment variables

# Optional: custom RustFS port (default 9050)
export RUSTFS_TEST_PORT=9050

# Optional: custom Vault port (default 8200)
export VAULT_TEST_PORT=8200

# Optional: enable verbose logging
export RUST_LOG=debug

Required binaries

Tests look for:

RustFS server – the binary verified by scripts/e2e_binary.py (default target/debug/rustfs)
vault – Vault CLI (must be on PATH)
/Users/dandan/Library/Python/3.9/bin/awscurl – AWS SigV4 helper

📋 Test Flow

Local backend

Prepare environment – create temporary directories and key storage paths
Start RustFS – launch the server with KMS enabled
Wait for readiness – confirm the port listener and S3 API
Configure KMS – send configuration via awscurl to the admin API
Start KMS – activate the KMS service
Exercise functionality
- Create a test bucket
- Run SSE-C encryption with client-provided keys
- Validate encryption/decryption behavior
Cleanup – stop processes and remove temporary files

Vault backend

Launch Vault – start the dev-mode server
Configure Vault
- Enable the transit secrets engine
- Create the rustfs-master-key
Start RustFS – run the server with KMS enabled
Configure KMS – point RustFS at Vault (address, token, transit config, key path)
Exercise functionality – complete the encryption/decryption workflow
Cleanup – stop all services

🛠️ Troubleshooting

Common issues

Q: RustFS server failed to become ready

lsof -i :9050
kill -9 <PID>  # Free the port if necessary

Q: Vault fails to start

which vault
vault version

Q: awscurl authentication fails

ls /Users/dandan/Library/Python/3.9/bin/awscurl
# Or install elsewhere
pip install awscurl
which awscurl  # Update the path in tests accordingly

Q: Tests time out

RUST_LOG=debug python3 scripts/e2e_binary.py run -- cargo test -p e2e_test test_local_kms_end_to_end -- --nocapture

Debug tips

Enable verbose logs

RUST_LOG=rustfs_kms=debug,rustfs=info python3 scripts/e2e_binary.py run -- cargo test -p e2e_test kms -- --nocapture

Keep temporary files – comment out cleanup logic to inspect generated configs
Pause execution – add std::thread::sleep for manual inspection during tests

Monitor ports

netstat -an | grep 9050
curl http://127.0.0.1:9050/health/ready

📊 Coverage

Functional

✅ Dynamic KMS configuration
✅ Local and Vault backends
✅ AWS S3-compatible encryption APIs
✅ Key lifecycle management
✅ Error handling and recovery paths
✅ High-availability behavior

Encryption modes

✅ SSE-C (customer-provided)
✅ SSE-S3 (S3-managed)
✅ SSE-KMS (KMS-managed)

S3 operations

✅ Object upload/download (SSE-C)
🚧 Multipart uploads (pending AWS SDK fixes)
🚧 Object replication (pending AWS SDK fixes)
🚧 Bucket encryption defaults (pending AWS SDK fixes)

KMS API

✅ Basic key management (create/list)
🚧 Full key lifecycle (pending AWS SDK fixes)
🚧 Direct encrypt/decrypt (pending AWS SDK fixes)
🚧 Data key operations (pending AWS SDK fixes)
✅ Service lifecycle (configure/start/stop/status)

Authentication

✅ Vault token auth
🚧 Vault AppRole auth

🔄 CI Integration

Designed to run inside CI/CD pipelines:

- name: Run KMS E2E Tests
  run: |
    sudo apt-get update
    sudo apt-get install -y vault
    pip install awscurl

    python3 scripts/e2e_binary.py build
    python3 scripts/e2e_binary.py run -- cargo test -p e2e_test kms -- --nocapture --test-threads=1

📚 References

These suites ensure KMS stability and reliability, building confidence for production deployments.

README.md Unescape Escape

KMS End-to-End Tests

📁 Test Overview

kms_local_test.rs

kms_vault_test.rs

kms_comprehensive_test.rs

kms_integration_test.rs

🚀 Running Tests

Prerequisites

Run individual suites

Local backend

Vault backend

High availability

Comprehensive features (disabled)

Run all KMS suites

Run serially (avoid port conflicts)

🔧 Configuration

Environment variables

Required binaries

📋 Test Flow

Local backend

Vault backend

🛠️ Troubleshooting

Common issues

Debug tips

📊 Coverage

Functional

Encryption modes

S3 operations

KMS API

Authentication

🔄 CI Integration

📚 References

README.md

`kms_local_test.rs`

`kms_vault_test.rs`

`kms_comprehensive_test.rs`

`kms_integration_test.rs`