mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-16 01:48:21 +00:00
Zhengchao An
ebbcfa3ac2
* fix(tier): decrypt transitioned objects instead of serving their ciphertext A GET on a managed-SSE object that lifecycle had transitioned to a remote tier returned the ciphertext with the plaintext's Content-Length and no error: silent corruption on read-through, and worse than a failed request because nothing signals it. Restore of the same object failed server-side with IncompleteBody while POST ?restore still answered 200, so the object simply never came back and HEAD never showed an x-amz-restore marker. Both symptoms are one cause. The transitioned read path built its fetch through new_getobjectreader, which decides nothing about encryption: it derived the range from the parts table — whose sizes are PLAINTEXT sizes — then used that range to fetch the object's STORED bytes from the tier, and handed the stream to the caller without any decrypt transform. The GET therefore served the first plaintext-length bytes of ciphertext; the restore copy-back, which validates against the stored size, came up short by exactly the encryption overhead. The path now builds the same ReadPlan the local read path uses, so a single place decides how stored bytes map to requested bytes. ReadPlan gains a two-phase API — build_for_request to learn the storage coordinates before issuing the tier fetch, into_object_reader to wrap the returned stream — because the tier fetch has to be positioned before a stream exists. The encryption resolver reaches the path from InstanceContext, the same source the local read uses. A restore read additionally stops synthesizing a range from the part number. A restore serves the stored representation (restore_request_active already forces the Plain branch), so a plaintext-coordinate range would be reinterpreted as a storage range and truncate the payload by its encoding overhead. An explicit caller range is already in storage coordinates on that path and is still honored, which two existing tests pin. crates/e2e_test/src/kms/kms_ilm_sse_kms_test.rs drops its #[ignore]: the transition test now runs and asserts the plaintext round-trips byte-identically through transition, read-through and restore. The same file had its enforcement switch stuck at false from a control experiment; it is back to true, so the test again exercises what its name and module docs claim. Fixes #6025. Refs rustfs/backlog#1582, rustfs/backlog#1637. * test(tier): pass resolver to transitioned reader tests
KMS End-to-End Tests
This directory contains the integration suites used to validate the full RustFS KMS (Key Management Service) workflow.
📁 Test Overview
kms_local_test.rs
End-to-end coverage for the local KMS backend:
- Auto-start and configure the local backend
- Configure KMS through the dynamic configuration API
- Verify SSE-C (client-provided keys)
- Exercise S3-compatible encryption/decryption
- Validate key lifecycle management
kms_vault_test.rs
End-to-end coverage for the Vault backend:
- Launch a Vault dev server automatically
- Configure the transit engine and encryption keys
- Configure KMS via the dynamic configuration API
- Run the full Vault integration flow
- Validate token authentication and encryption operations
kms_comprehensive_test.rs
Full KMS capability suite (currently disabled because of AWS SDK compatibility issues):
- Bucket encryption configuration: SSE-S3 and SSE-KMS defaults
- All SSE encryption modes:
- SSE-S3 (S3-managed server-side encryption)
- SSE-KMS (KMS-managed server-side encryption)
- SSE-C (client-provided keys)
- Object operations: upload, download, and validation for every SSE mode
- Multipart uploads: cover each SSE mode
- Object replication: cross-mode replication scenarios
- Complete KMS API management:
- Key lifecycle (create, list, describe, delete, cancel delete)
- Direct encrypt/decrypt operations
- Data key generation and handling
- KMS service lifecycle (start, stop, status)
kms_integration_test.rs
Broad integration tests that exercise:
- Multiple backends
- KMS lifecycle management
- Error handling and recovery
- Note: currently disabled because of AWS SDK compatibility gaps
🚀 Running Tests
Prerequisites
-
System dependencies
# macOS brew install vault awscurl # Ubuntu/Debian apt-get install vault pip install awscurl -
Build RustFS
cargo build
Run individual suites
Local backend
cd crates/e2e_test
cargo test test_local_kms_end_to_end -- --nocapture
Vault backend
cd crates/e2e_test
cargo test test_vault_kms_end_to_end -- --nocapture
High availability
cd crates/e2e_test
cargo test test_vault_kms_high_availability -- --nocapture
Comprehensive features (disabled)
cd crates/e2e_test
# Disabled due to AWS SDK compatibility gaps
# cargo test test_comprehensive_kms_functionality -- --nocapture
# cargo test test_sse_modes_compatibility -- --nocapture
# cargo test test_kms_api_comprehensive -- --nocapture
Run all KMS suites
cd crates/e2e_test
cargo test kms -- --nocapture
Run serially (avoid port conflicts)
cd crates/e2e_test
cargo test kms -- --nocapture --test-threads=1
🔧 Configuration
Environment variables
# Optional: custom RustFS port (default 9050)
export RUSTFS_TEST_PORT=9050
# Optional: custom Vault port (default 8200)
export VAULT_TEST_PORT=8200
# Optional: enable verbose logging
export RUST_LOG=debug
Required binaries
Tests look for:
../../target/debug/rustfs– RustFS servervault– Vault CLI (must be on PATH)/Users/dandan/Library/Python/3.9/bin/awscurl– AWS SigV4 helper
📋 Test Flow
Local backend
- Prepare environment – create temporary directories and key storage paths
- Start RustFS – launch the server with KMS enabled
- Wait for readiness – confirm the port listener and S3 API
- Configure KMS – send configuration via awscurl to the admin API
- Start KMS – activate the KMS service
- Exercise functionality
- Create a test bucket
- Run SSE-C encryption with client-provided keys
- Validate encryption/decryption behavior
- Cleanup – stop processes and remove temporary files
Vault backend
- Launch Vault – start the dev-mode server
- Configure Vault
- Enable the transit secrets engine
- Create the
rustfs-master-key
- Start RustFS – run the server with KMS enabled
- Configure KMS – point RustFS at Vault (address, token, transit config, key path)
- Exercise functionality – complete the encryption/decryption workflow
- Cleanup – stop all services
🛠️ Troubleshooting
Common issues
Q: RustFS server failed to become ready
lsof -i :9050
kill -9 <PID> # Free the port if necessary
Q: Vault fails to start
which vault
vault version
Q: awscurl authentication fails
ls /Users/dandan/Library/Python/3.9/bin/awscurl
# Or install elsewhere
pip install awscurl
which awscurl # Update the path in tests accordingly
Q: Tests time out
RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
Debug tips
-
Enable verbose logs
RUST_LOG=rustfs_kms=debug,rustfs=info cargo test -- --nocapture -
Keep temporary files – comment out cleanup logic to inspect generated configs
-
Pause execution – add
std::thread::sleepfor manual inspection during tests -
Monitor ports
netstat -an | grep 9050 curl http://127.0.0.1:9050/health/ready
📊 Coverage
Functional
- ✅ Dynamic KMS configuration
- ✅ Local and Vault backends
- ✅ AWS S3-compatible encryption APIs
- ✅ Key lifecycle management
- ✅ Error handling and recovery paths
- ✅ High-availability behavior
Encryption modes
- ✅ SSE-C (customer-provided)
- ✅ SSE-S3 (S3-managed)
- ✅ SSE-KMS (KMS-managed)
S3 operations
- ✅ Object upload/download (SSE-C)
- 🚧 Multipart uploads (pending AWS SDK fixes)
- 🚧 Object replication (pending AWS SDK fixes)
- 🚧 Bucket encryption defaults (pending AWS SDK fixes)
KMS API
- ✅ Basic key management (create/list)
- 🚧 Full key lifecycle (pending AWS SDK fixes)
- 🚧 Direct encrypt/decrypt (pending AWS SDK fixes)
- 🚧 Data key operations (pending AWS SDK fixes)
- ✅ Service lifecycle (configure/start/stop/status)
Authentication
- ✅ Vault token auth
- 🚧 Vault AppRole auth
🔄 CI Integration
Designed to run inside CI/CD pipelines:
- name: Run KMS E2E Tests
run: |
sudo apt-get update
sudo apt-get install -y vault
pip install awscurl
cargo build
cd crates/e2e_test
cargo test kms -- --nocapture --test-threads=1
📚 References
These suites ensure KMS stability and reliability, building confidence for production deployments.