mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-11 07:36:53 +00:00
fix: restore localized samples in tests (#749)
* fix: restore required localized examples * style: fix formatting issues
This commit is contained in:
+130
-144
@@ -1,267 +1,253 @@
|
||||
# KMS End-to-End Tests
|
||||
|
||||
本目录包含 RustFS KMS (Key Management Service) 的端到端集成测试,用于验证完整的 KMS 功能流程。
|
||||
This directory contains the integration suites used to validate the full RustFS KMS (Key Management Service) workflow.
|
||||
|
||||
## 📁 测试文件说明
|
||||
## 📁 Test Overview
|
||||
|
||||
### `kms_local_test.rs`
|
||||
本地KMS后端的端到端测试,包含:
|
||||
- 自动启动和配置本地KMS后端
|
||||
- 通过动态配置API配置KMS服务
|
||||
- 测试SSE-C(客户端提供密钥)加密流程
|
||||
- 验证S3兼容的对象加密/解密操作
|
||||
- 密钥生命周期管理测试
|
||||
End-to-end coverage for the local KMS backend:
|
||||
- Auto-start and configure the local backend
|
||||
- Configure KMS through the dynamic configuration API
|
||||
- Verify SSE-C (client-provided keys)
|
||||
- Exercise S3-compatible encryption/decryption
|
||||
- Validate key lifecycle management
|
||||
|
||||
### `kms_vault_test.rs`
|
||||
Vault KMS后端的端到端测试,包含:
|
||||
- 自动启动Vault开发服务器
|
||||
- 配置Vault transit engine和密钥
|
||||
- 通过动态配置API配置KMS服务
|
||||
- 测试完整的Vault KMS集成
|
||||
- 验证Token认证和加密操作
|
||||
End-to-end coverage for the Vault backend:
|
||||
- Launch a Vault dev server automatically
|
||||
- Configure the transit engine and encryption keys
|
||||
- Configure KMS via the dynamic configuration API
|
||||
- Run the full Vault integration flow
|
||||
- Validate token authentication and encryption operations
|
||||
|
||||
### `kms_comprehensive_test.rs`
|
||||
**完整的KMS功能测试套件**(当前因AWS SDK API兼容性问题暂时禁用),包含:
|
||||
- **Bucket加密配置**: SSE-S3和SSE-KMS默认加密设置
|
||||
- **完整的SSE加密模式测试**:
|
||||
- SSE-S3: S3管理的服务端加密
|
||||
- SSE-KMS: KMS管理的服务端加密
|
||||
- SSE-C: 客户端提供密钥的服务端加密
|
||||
- **对象操作测试**: 上传、下载、验证三种SSE模式
|
||||
- **分片上传测试**: 多部分上传支持所有SSE模式
|
||||
- **对象复制测试**: 不同SSE模式间的复制操作
|
||||
- **完整KMS API管理**:
|
||||
- 密钥生命周期管理(创建、列表、描述、删除、取消删除)
|
||||
- 直接加密/解密操作
|
||||
- 数据密钥生成和操作
|
||||
- KMS服务管理(启动、停止、状态查询)
|
||||
**Full KMS capability suite** (currently disabled because of AWS SDK compatibility issues):
|
||||
- **Bucket encryption configuration**: SSE-S3 and SSE-KMS defaults
|
||||
- **All SSE encryption modes**:
|
||||
- SSE-S3 (S3-managed server-side encryption)
|
||||
- SSE-KMS (KMS-managed server-side encryption)
|
||||
- SSE-C (client-provided keys)
|
||||
- **Object operations**: upload, download, and validation for every SSE mode
|
||||
- **Multipart uploads**: cover each SSE mode
|
||||
- **Object replication**: cross-mode replication scenarios
|
||||
- **Complete KMS API management**:
|
||||
- Key lifecycle (create, list, describe, delete, cancel delete)
|
||||
- Direct encrypt/decrypt operations
|
||||
- Data key generation and handling
|
||||
- KMS service lifecycle (start, stop, status)
|
||||
|
||||
### `kms_integration_test.rs`
|
||||
综合性KMS集成测试,包含:
|
||||
- 多后端兼容性测试
|
||||
- KMS服务生命周期测试
|
||||
- 错误处理和恢复测试
|
||||
- **注意**: 当前因AWS SDK API兼容性问题暂时禁用
|
||||
Broad integration tests that exercise:
|
||||
- Multiple backends
|
||||
- KMS lifecycle management
|
||||
- Error handling and recovery
|
||||
- **Note**: currently disabled because of AWS SDK compatibility gaps
|
||||
|
||||
## 🚀 如何运行测试
|
||||
## 🚀 Running Tests
|
||||
|
||||
### 前提条件
|
||||
### Prerequisites
|
||||
|
||||
1. **系统依赖**:
|
||||
1. **System dependencies**
|
||||
```bash
|
||||
# macOS
|
||||
brew install vault awscurl
|
||||
|
||||
|
||||
# Ubuntu/Debian
|
||||
apt-get install vault
|
||||
pip install awscurl
|
||||
```
|
||||
|
||||
2. **构建RustFS**:
|
||||
2. **Build RustFS**
|
||||
```bash
|
||||
# 在项目根目录
|
||||
cargo build
|
||||
```
|
||||
|
||||
### 运行单个测试
|
||||
### Run individual suites
|
||||
|
||||
#### 本地KMS测试
|
||||
#### Local backend
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
cargo test test_local_kms_end_to_end -- --nocapture
|
||||
```
|
||||
|
||||
#### Vault KMS测试
|
||||
#### Vault backend
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
cargo test test_vault_kms_end_to_end -- --nocapture
|
||||
```
|
||||
|
||||
#### 高可用性测试
|
||||
#### High availability
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
cargo test test_vault_kms_high_availability -- --nocapture
|
||||
```
|
||||
|
||||
#### 完整功能测试(开发中)
|
||||
#### Comprehensive features (disabled)
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
# 注意:以下测试因AWS SDK API兼容性问题暂时禁用
|
||||
# Disabled due to AWS SDK compatibility gaps
|
||||
# cargo test test_comprehensive_kms_functionality -- --nocapture
|
||||
# cargo test test_sse_modes_compatibility -- --nocapture
|
||||
# cargo test test_sse_modes_compatibility -- --nocapture
|
||||
# cargo test test_kms_api_comprehensive -- --nocapture
|
||||
```
|
||||
|
||||
### 运行所有KMS测试
|
||||
### Run all KMS suites
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
cargo test kms -- --nocapture
|
||||
```
|
||||
|
||||
### 串行运行(避免端口冲突)
|
||||
### Run serially (avoid port conflicts)
|
||||
```bash
|
||||
cd crates/e2e_test
|
||||
cargo test kms -- --nocapture --test-threads=1
|
||||
```
|
||||
|
||||
## 🔧 测试配置
|
||||
## 🔧 Configuration
|
||||
|
||||
### 环境变量
|
||||
### Environment variables
|
||||
```bash
|
||||
# 可选:自定义端口(默认使用9050)
|
||||
# Optional: custom RustFS port (default 9050)
|
||||
export RUSTFS_TEST_PORT=9050
|
||||
|
||||
# 可选:自定义Vault端口(默认使用8200)
|
||||
# Optional: custom Vault port (default 8200)
|
||||
export VAULT_TEST_PORT=8200
|
||||
|
||||
# 可选:启用详细日志
|
||||
# Optional: enable verbose logging
|
||||
export RUST_LOG=debug
|
||||
```
|
||||
|
||||
### 依赖的二进制文件路径
|
||||
### Required binaries
|
||||
|
||||
测试会自动查找以下二进制文件:
|
||||
- `../../target/debug/rustfs` - RustFS服务器
|
||||
- `vault` - Vault (需要在PATH中)
|
||||
- `/Users/dandan/Library/Python/3.9/bin/awscurl` - AWS签名工具
|
||||
Tests look for:
|
||||
- `../../target/debug/rustfs` – RustFS server
|
||||
- `vault` – Vault CLI (must be on PATH)
|
||||
- `/Users/dandan/Library/Python/3.9/bin/awscurl` – AWS SigV4 helper
|
||||
|
||||
## 📋 测试流程说明
|
||||
## 📋 Test Flow
|
||||
|
||||
### Local KMS测试流程
|
||||
1. **环境准备**:创建临时目录,设置KMS密钥存储路径
|
||||
2. **启动服务**:启动RustFS服务器,启用KMS功能
|
||||
3. **等待就绪**:检查端口监听和S3 API响应
|
||||
4. **配置KMS**:通过awscurl发送配置请求到admin API
|
||||
5. **启动KMS**:激活KMS服务
|
||||
6. **功能测试**:
|
||||
- 创建测试存储桶
|
||||
- 测试SSE-C加密(客户端提供密钥)
|
||||
- 验证对象加密/解密
|
||||
7. **清理**:终止进程,清理临时文件
|
||||
### Local backend
|
||||
1. **Prepare environment** – create temporary directories and key storage paths
|
||||
2. **Start RustFS** – launch the server with KMS enabled
|
||||
3. **Wait for readiness** – confirm the port listener and S3 API
|
||||
4. **Configure KMS** – send configuration via awscurl to the admin API
|
||||
5. **Start KMS** – activate the KMS service
|
||||
6. **Exercise functionality**
|
||||
- Create a test bucket
|
||||
- Run SSE-C encryption with client-provided keys
|
||||
- Validate encryption/decryption behavior
|
||||
7. **Cleanup** – stop processes and remove temporary files
|
||||
|
||||
### Vault KMS测试流程
|
||||
1. **启动Vault**:使用开发模式启动Vault服务器
|
||||
2. **配置Vault**:
|
||||
- 启用transit secrets engine
|
||||
- 创建加密密钥(rustfs-master-key)
|
||||
3. **启动RustFS**:启用KMS功能的RustFS服务器
|
||||
4. **配置KMS**:通过API配置Vault后端,包含:
|
||||
- Vault地址和Token认证
|
||||
- Transit engine配置
|
||||
- 密钥路径设置
|
||||
5. **功能测试**:完整的加密/解密流程测试
|
||||
6. **清理**:终止所有进程
|
||||
### Vault backend
|
||||
1. **Launch Vault** – start the dev-mode server
|
||||
2. **Configure Vault**
|
||||
- Enable the transit secrets engine
|
||||
- Create the `rustfs-master-key`
|
||||
3. **Start RustFS** – run the server with KMS enabled
|
||||
4. **Configure KMS** – point RustFS at Vault (address, token, transit config, key path)
|
||||
5. **Exercise functionality** – complete the encryption/decryption workflow
|
||||
6. **Cleanup** – stop all services
|
||||
|
||||
## 🛠️ 故障排除
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### 常见问题
|
||||
### Common issues
|
||||
|
||||
**Q: 测试失败 "RustFS server failed to become ready"**
|
||||
```
|
||||
A: 检查端口是否被占用:
|
||||
**Q: `RustFS server failed to become ready`**
|
||||
```bash
|
||||
lsof -i :9050
|
||||
kill -9 <PID> # 如果有进程占用端口
|
||||
kill -9 <PID> # Free the port if necessary
|
||||
```
|
||||
|
||||
**Q: Vault服务启动失败**
|
||||
```
|
||||
A: 确保Vault已安装且在PATH中:
|
||||
**Q: Vault fails to start**
|
||||
```bash
|
||||
which vault
|
||||
vault version
|
||||
```
|
||||
|
||||
**Q: awscurl认证失败**
|
||||
```
|
||||
A: 检查awscurl路径是否正确:
|
||||
**Q: awscurl authentication fails**
|
||||
```bash
|
||||
ls /Users/dandan/Library/Python/3.9/bin/awscurl
|
||||
# 或安装到不同路径:
|
||||
# Or install elsewhere
|
||||
pip install awscurl
|
||||
which awscurl # 然后更新测试中的路径
|
||||
which awscurl # Update the path in tests accordingly
|
||||
```
|
||||
|
||||
**Q: 测试超时**
|
||||
```
|
||||
A: 增加等待时间或检查日志:
|
||||
**Q: Tests time out**
|
||||
```bash
|
||||
RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
|
||||
```
|
||||
|
||||
### 调试技巧
|
||||
### Debug tips
|
||||
|
||||
1. **查看详细日志**:
|
||||
1. **Enable verbose logs**
|
||||
```bash
|
||||
RUST_LOG=rustfs_kms=debug,rustfs=info cargo test -- --nocapture
|
||||
```
|
||||
|
||||
2. **保留临时文件**:
|
||||
修改测试代码,注释掉清理部分,检查生成的配置文件
|
||||
2. **Keep temporary files** – comment out cleanup logic to inspect generated configs
|
||||
|
||||
3. **单步调试**:
|
||||
在测试中添加 `std::thread::sleep` 来暂停执行,手动检查服务状态
|
||||
3. **Pause execution** – add `std::thread::sleep` for manual inspection during tests
|
||||
|
||||
4. **端口检查**:
|
||||
4. **Monitor ports**
|
||||
```bash
|
||||
# 测试运行时检查端口状态
|
||||
netstat -an | grep 9050
|
||||
curl http://127.0.0.1:9050/minio/health/ready
|
||||
```
|
||||
|
||||
## 📊 测试覆盖范围
|
||||
## 📊 Coverage
|
||||
|
||||
### 功能覆盖
|
||||
- ✅ KMS服务动态配置
|
||||
- ✅ 本地和Vault后端支持
|
||||
- ✅ AWS S3兼容加密接口
|
||||
- ✅ 密钥管理和生命周期
|
||||
- ✅ 错误处理和恢复
|
||||
- ✅ 高可用性场景
|
||||
### Functional
|
||||
- ✅ Dynamic KMS configuration
|
||||
- ✅ Local and Vault backends
|
||||
- ✅ AWS S3-compatible encryption APIs
|
||||
- ✅ Key lifecycle management
|
||||
- ✅ Error handling and recovery paths
|
||||
- ✅ High-availability behavior
|
||||
|
||||
### 加密模式覆盖
|
||||
- ✅ SSE-C (Server-Side Encryption with Customer-Provided Keys)
|
||||
- ✅ SSE-S3 (Server-Side Encryption with S3-Managed Keys)
|
||||
- ✅ SSE-KMS (Server-Side Encryption with KMS-Managed Keys)
|
||||
### Encryption modes
|
||||
- ✅ SSE-C (customer-provided)
|
||||
- ✅ SSE-S3 (S3-managed)
|
||||
- ✅ SSE-KMS (KMS-managed)
|
||||
|
||||
### S3操作覆盖
|
||||
- ✅ 对象上传/下载 (SSE-C模式)
|
||||
- 🚧 分片上传 (需要AWS SDK兼容性修复)
|
||||
- 🚧 对象复制 (需要AWS SDK兼容性修复)
|
||||
- 🚧 Bucket加密配置 (需要AWS SDK兼容性修复)
|
||||
### S3 operations
|
||||
- ✅ Object upload/download (SSE-C)
|
||||
- 🚧 Multipart uploads (pending AWS SDK fixes)
|
||||
- 🚧 Object replication (pending AWS SDK fixes)
|
||||
- 🚧 Bucket encryption defaults (pending AWS SDK fixes)
|
||||
|
||||
### KMS API覆盖
|
||||
- ✅ 基础密钥管理 (创建、列表)
|
||||
- 🚧 完整密钥生命周期 (需要AWS SDK兼容性修复)
|
||||
- 🚧 直接加密/解密操作 (需要AWS SDK兼容性修复)
|
||||
- 🚧 数据密钥生成和解密 (需要AWS SDK兼容性修复)
|
||||
- ✅ KMS服务管理 (配置、启动、停止、状态)
|
||||
### KMS API
|
||||
- ✅ Basic key management (create/list)
|
||||
- 🚧 Full key lifecycle (pending AWS SDK fixes)
|
||||
- 🚧 Direct encrypt/decrypt (pending AWS SDK fixes)
|
||||
- 🚧 Data key operations (pending AWS SDK fixes)
|
||||
- ✅ Service lifecycle (configure/start/stop/status)
|
||||
|
||||
### 认证方式覆盖
|
||||
- ✅ Vault Token认证
|
||||
- 🚧 Vault AppRole认证
|
||||
### Authentication
|
||||
- ✅ Vault token auth
|
||||
- 🚧 Vault AppRole auth
|
||||
|
||||
## 🔄 持续集成
|
||||
## 🔄 CI Integration
|
||||
|
||||
这些测试设计为可在CI/CD环境中运行:
|
||||
Designed to run inside CI/CD pipelines:
|
||||
|
||||
```yaml
|
||||
# GitHub Actions 示例
|
||||
- name: Run KMS E2E Tests
|
||||
run: |
|
||||
# 安装依赖
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y vault
|
||||
pip install awscurl
|
||||
|
||||
# 构建并测试
|
||||
|
||||
cargo build
|
||||
cd crates/e2e_test
|
||||
cargo test kms -- --nocapture --test-threads=1
|
||||
```
|
||||
|
||||
## 📚 相关文档
|
||||
## 📚 References
|
||||
|
||||
- [KMS 配置文档](../../../../docs/kms/README.md) - KMS功能完整文档
|
||||
- [动态配置API](../../../../docs/kms/http-api.md) - REST API接口说明
|
||||
- [故障排除指南](../../../../docs/kms/troubleshooting.md) - 常见问题解决
|
||||
- [KMS configuration guide](../../../../docs/kms/README.md)
|
||||
- [Dynamic configuration API](../../../../docs/kms/http-api.md)
|
||||
- [Troubleshooting](../../../../docs/kms/troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
*这些测试确保KMS功能的稳定性和可靠性,为生产环境部署提供信心。*
|
||||
*These suites ensure KMS stability and reliability, building confidence for production deployments.*
|
||||
|
||||
@@ -547,9 +547,9 @@ pub async fn test_multipart_upload_with_config(
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let total_size = config.total_size();
|
||||
|
||||
info!("🧪 开始分片上传测试 - {:?}", config.encryption_type);
|
||||
info!("🧪 Starting multipart upload test - {:?}", config.encryption_type);
|
||||
info!(
|
||||
" 对象: {}, 分片: {}个, 每片: {}MB, 总计: {}MB",
|
||||
" Object: {}, parts: {}, part size: {} MB, total: {} MB",
|
||||
config.object_key,
|
||||
config.total_parts,
|
||||
config.part_size / (1024 * 1024),
|
||||
@@ -589,7 +589,7 @@ pub async fn test_multipart_upload_with_config(
|
||||
|
||||
let create_multipart_output = create_request.send().await?;
|
||||
let upload_id = create_multipart_output.upload_id().unwrap();
|
||||
info!("📋 创建分片上传,ID: {}", upload_id);
|
||||
info!("📋 Created multipart upload, ID: {}", upload_id);
|
||||
|
||||
// Step 2: Upload parts
|
||||
let mut completed_parts = Vec::new();
|
||||
@@ -598,7 +598,7 @@ pub async fn test_multipart_upload_with_config(
|
||||
let end = std::cmp::min(start + config.part_size, total_size);
|
||||
let part_data = &test_data[start..end];
|
||||
|
||||
info!("📤 上传分片 {} ({:.2}MB)", part_number, part_data.len() as f64 / (1024.0 * 1024.0));
|
||||
info!("📤 Uploading part {} ({:.2} MB)", part_number, part_data.len() as f64 / (1024.0 * 1024.0));
|
||||
|
||||
let mut upload_request = s3_client
|
||||
.upload_part()
|
||||
@@ -625,7 +625,7 @@ pub async fn test_multipart_upload_with_config(
|
||||
.build(),
|
||||
);
|
||||
|
||||
debug!("分片 {} 上传完成,ETag: {}", part_number, etag);
|
||||
debug!("Part {} uploaded with ETag {}", part_number, etag);
|
||||
}
|
||||
|
||||
// Step 3: Complete multipart upload
|
||||
@@ -633,7 +633,7 @@ pub async fn test_multipart_upload_with_config(
|
||||
.set_parts(Some(completed_parts))
|
||||
.build();
|
||||
|
||||
info!("🔗 完成分片上传");
|
||||
info!("🔗 Completing multipart upload");
|
||||
let complete_output = s3_client
|
||||
.complete_multipart_upload()
|
||||
.bucket(bucket)
|
||||
@@ -643,10 +643,10 @@ pub async fn test_multipart_upload_with_config(
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
debug!("完成分片上传,ETag: {:?}", complete_output.e_tag());
|
||||
debug!("Multipart upload finalized with ETag {:?}", complete_output.e_tag());
|
||||
|
||||
// Step 4: Download and verify
|
||||
info!("📥 下载文件并验证");
|
||||
info!("📥 Downloading object for verification");
|
||||
let mut get_request = s3_client.get_object().bucket(bucket).key(&config.object_key);
|
||||
|
||||
// Add encryption headers for SSE-C GET
|
||||
@@ -680,7 +680,7 @@ pub async fn test_multipart_upload_with_config(
|
||||
assert_eq!(downloaded_data.len(), total_size);
|
||||
assert_eq!(&downloaded_data[..], &test_data[..]);
|
||||
|
||||
info!("✅ 分片上传测试通过 - {:?}", config.encryption_type);
|
||||
info!("✅ Multipart upload test passed - {:?}", config.encryption_type);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -700,7 +700,7 @@ pub async fn test_all_multipart_encryption_types(
|
||||
bucket: &str,
|
||||
base_object_key: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
info!("🧪 测试所有加密类型的分片上传");
|
||||
info!("🧪 Testing multipart uploads for every encryption type");
|
||||
|
||||
let part_size = 5 * 1024 * 1024; // 5MB per part
|
||||
let total_parts = 2;
|
||||
@@ -718,7 +718,7 @@ pub async fn test_all_multipart_encryption_types(
|
||||
test_multipart_upload_with_config(s3_client, bucket, &config).await?;
|
||||
}
|
||||
|
||||
info!("✅ 所有加密类型的分片上传测试通过");
|
||||
info!("✅ Multipart uploads succeeded for every encryption type");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -201,7 +201,7 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
|
||||
let total_parts = 2;
|
||||
let total_size = part_size * total_parts;
|
||||
|
||||
// 生成测试数据
|
||||
// Generate test data
|
||||
let test_data: Vec<u8> = (0..total_size).map(|i| ((i / 1000) % 256) as u8).collect();
|
||||
|
||||
info!(
|
||||
@@ -275,43 +275,43 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
debug!("完成加密分片上传,ETag: {:?}", complete_output.e_tag());
|
||||
debug!("Encrypted multipart upload completed with ETag {:?}", complete_output.e_tag());
|
||||
|
||||
// 步骤 4:HEAD 请求检查元数据
|
||||
info!("📋 检查对象元数据");
|
||||
// Step 4: HEAD request to inspect metadata
|
||||
info!("📋 Inspecting object metadata");
|
||||
let head_response = s3_client.head_object().bucket(TEST_BUCKET).key(object_key).send().await?;
|
||||
|
||||
debug!("HEAD 响应 SSE: {:?}", head_response.server_side_encryption());
|
||||
debug!("HEAD 响应 元数据:{:?}", head_response.metadata());
|
||||
debug!("HEAD response SSE: {:?}", head_response.server_side_encryption());
|
||||
debug!("HEAD response metadata: {:?}", head_response.metadata());
|
||||
|
||||
// 步骤 5:GET 请求下载并验证
|
||||
info!("📥 下载加密文件并验证");
|
||||
// Step 5: GET request to download and verify
|
||||
info!("📥 Downloading encrypted object for verification");
|
||||
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
|
||||
|
||||
debug!("GET 响应 SSE: {:?}", get_response.server_side_encryption());
|
||||
debug!("GET response SSE: {:?}", get_response.server_side_encryption());
|
||||
|
||||
// 🎯 关键验证:GET 响应必须包含 SSE-S3 加密头
|
||||
// 🎯 Critical check: GET response must include SSE-S3 headers
|
||||
assert_eq!(
|
||||
get_response.server_side_encryption(),
|
||||
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
|
||||
);
|
||||
|
||||
// 验证数据完整性
|
||||
// Verify data integrity
|
||||
let downloaded_data = get_response.body.collect().await?.into_bytes();
|
||||
assert_eq!(downloaded_data.len(), total_size);
|
||||
assert_eq!(&downloaded_data[..], &test_data[..]);
|
||||
|
||||
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
|
||||
info!("✅ 步骤 3 通过:分片上传 + SSE-S3 加密功能正常");
|
||||
info!("✅ Step 3 passed: multipart upload with SSE-S3 encryption");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 步骤 4:测试更大的分片上传(测试流式加密)
|
||||
/// Step 4: test larger multipart uploads (streaming encryption)
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
init_logging();
|
||||
info!("🧪 步骤 4:测试大文件分片上传加密");
|
||||
info!("🧪 Step 4: test large-file multipart encryption");
|
||||
|
||||
let mut kms_env = LocalKMSTestEnvironment::new().await?;
|
||||
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
|
||||
@@ -321,18 +321,18 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
kms_env.base_env.create_test_bucket(TEST_BUCKET).await?;
|
||||
|
||||
let object_key = "test-large-multipart-encrypted";
|
||||
let part_size = 6 * 1024 * 1024; // 6MB per part (大于 1MB 加密块大小)
|
||||
let total_parts = 3; // 总共 18MB
|
||||
let part_size = 6 * 1024 * 1024; // 6 MB per part (greater than the 1 MB encryption chunk)
|
||||
let total_parts = 3; // 18 MB total
|
||||
let total_size = part_size * total_parts;
|
||||
|
||||
info!(
|
||||
"🗂️ 生成大文件测试数据:{} parts,每个 {}MB,总计 {}MB",
|
||||
"🗂️ Generated large-file test data: {} parts, {} MB each, {} MB total",
|
||||
total_parts,
|
||||
part_size / (1024 * 1024),
|
||||
total_size / (1024 * 1024)
|
||||
);
|
||||
|
||||
// 生成大文件测试数据(使用复杂模式便于验证)
|
||||
// Generate large test data (complex pattern for validation)
|
||||
let test_data: Vec<u8> = (0..total_size)
|
||||
.map(|i| {
|
||||
let part_num = i / part_size;
|
||||
@@ -341,9 +341,9 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
})
|
||||
.collect();
|
||||
|
||||
info!("🔐 开始大文件分片上传(SSE-S3 加密)");
|
||||
info!("🔐 Starting large-file multipart upload (SSE-S3 encryption)");
|
||||
|
||||
// 创建分片上传
|
||||
// Create multipart upload
|
||||
let create_multipart_output = s3_client
|
||||
.create_multipart_upload()
|
||||
.bucket(TEST_BUCKET)
|
||||
@@ -353,9 +353,9 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
.await?;
|
||||
|
||||
let upload_id = create_multipart_output.upload_id().unwrap();
|
||||
info!("📋 创建大文件加密分片上传,ID: {}", upload_id);
|
||||
info!("📋 Created large encrypted multipart upload, ID: {}", upload_id);
|
||||
|
||||
// 上传各个分片
|
||||
// Upload each part
|
||||
let mut completed_parts = Vec::new();
|
||||
for part_number in 1..=total_parts {
|
||||
let start = (part_number - 1) * part_size;
|
||||
@@ -363,7 +363,7 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
let part_data = &test_data[start..end];
|
||||
|
||||
info!(
|
||||
"🔐 上传大文件加密分片 {} ({:.2}MB)",
|
||||
"🔐 Uploading encrypted large-file part {} ({:.2} MB)",
|
||||
part_number,
|
||||
part_data.len() as f64 / (1024.0 * 1024.0)
|
||||
);
|
||||
@@ -386,15 +386,15 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
.build(),
|
||||
);
|
||||
|
||||
debug!("大文件加密分片 {} 上传完成,ETag: {}", part_number, etag);
|
||||
debug!("Large encrypted part {} uploaded with ETag {}", part_number, etag);
|
||||
}
|
||||
|
||||
// 完成分片上传
|
||||
// Complete the multipart upload
|
||||
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
|
||||
.set_parts(Some(completed_parts))
|
||||
.build();
|
||||
|
||||
info!("🔗 完成大文件加密分片上传");
|
||||
info!("🔗 Completing large encrypted multipart upload");
|
||||
let complete_output = s3_client
|
||||
.complete_multipart_upload()
|
||||
.bucket(TEST_BUCKET)
|
||||
@@ -404,40 +404,40 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
debug!("完成大文件加密分片上传,ETag: {:?}", complete_output.e_tag());
|
||||
debug!("Large encrypted multipart upload completed with ETag {:?}", complete_output.e_tag());
|
||||
|
||||
// 下载并验证
|
||||
info!("📥 下载大文件并验证");
|
||||
// Download and verify
|
||||
info!("📥 Downloading large object for verification");
|
||||
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
|
||||
|
||||
// 验证加密头
|
||||
// Verify encryption headers
|
||||
assert_eq!(
|
||||
get_response.server_side_encryption(),
|
||||
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
|
||||
);
|
||||
|
||||
// 验证数据完整性
|
||||
// Verify data integrity
|
||||
let downloaded_data = get_response.body.collect().await?.into_bytes();
|
||||
assert_eq!(downloaded_data.len(), total_size);
|
||||
|
||||
// 逐字节验证数据(对于大文件更严格)
|
||||
// Validate bytes individually (stricter for large files)
|
||||
for (i, (&actual, &expected)) in downloaded_data.iter().zip(test_data.iter()).enumerate() {
|
||||
if actual != expected {
|
||||
panic!("大文件数据在第{i}字节不匹配:实际={actual}, 期待={expected}");
|
||||
panic!("Large file mismatch at byte {i}: actual={actual}, expected={expected}");
|
||||
}
|
||||
}
|
||||
|
||||
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
|
||||
info!("✅ 步骤 4 通过:大文件分片上传加密功能正常");
|
||||
info!("✅ Step 4 passed: large-file multipart encryption succeeded");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// 步骤 5:测试所有加密类型的分片上传
|
||||
/// Step 5: test multipart uploads for every encryption mode
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
init_logging();
|
||||
info!("🧪 步骤 5:测试所有加密类型的分片上传");
|
||||
info!("🧪 Step 5: test multipart uploads for every encryption mode");
|
||||
|
||||
let mut kms_env = LocalKMSTestEnvironment::new().await?;
|
||||
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
|
||||
@@ -450,8 +450,8 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
|
||||
let total_parts = 2;
|
||||
let total_size = part_size * total_parts;
|
||||
|
||||
// 测试 SSE-KMS
|
||||
info!("🔐 测试 SSE-KMS 分片上传");
|
||||
// Test SSE-KMS
|
||||
info!("🔐 Testing SSE-KMS multipart upload");
|
||||
test_multipart_encryption_type(
|
||||
&s3_client,
|
||||
TEST_BUCKET,
|
||||
@@ -463,8 +463,8 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
|
||||
)
|
||||
.await?;
|
||||
|
||||
// 测试 SSE-C
|
||||
info!("🔐 测试 SSE-C 分片上传");
|
||||
// Test SSE-C
|
||||
info!("🔐 Testing SSE-C multipart upload");
|
||||
test_multipart_encryption_type(
|
||||
&s3_client,
|
||||
TEST_BUCKET,
|
||||
@@ -477,7 +477,7 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
|
||||
.await?;
|
||||
|
||||
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
|
||||
info!("✅ 步骤 5 通过:所有加密类型的分片上传功能正常");
|
||||
info!("✅ Step 5 passed: multipart uploads succeeded for every encryption mode");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -487,7 +487,7 @@ enum EncryptionType {
|
||||
SSEC,
|
||||
}
|
||||
|
||||
/// 辅助函数:测试特定加密类型的分片上传
|
||||
/// Helper: test multipart uploads for a specific encryption type
|
||||
async fn test_multipart_encryption_type(
|
||||
s3_client: &aws_sdk_s3::Client,
|
||||
bucket: &str,
|
||||
@@ -497,10 +497,10 @@ async fn test_multipart_encryption_type(
|
||||
total_parts: usize,
|
||||
encryption_type: EncryptionType,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
// 生成测试数据
|
||||
// Generate test data
|
||||
let test_data: Vec<u8> = (0..total_size).map(|i| ((i * 7) % 256) as u8).collect();
|
||||
|
||||
// 准备 SSE-C 所需的密钥(如果需要)
|
||||
// Prepare SSE-C keys when required
|
||||
let (sse_c_key, sse_c_md5) = if matches!(encryption_type, EncryptionType::SSEC) {
|
||||
let key = "01234567890123456789012345678901";
|
||||
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
|
||||
@@ -510,9 +510,9 @@ async fn test_multipart_encryption_type(
|
||||
(None, None)
|
||||
};
|
||||
|
||||
info!("📋 创建分片上传 - {:?}", encryption_type);
|
||||
info!("📋 Creating multipart upload - {:?}", encryption_type);
|
||||
|
||||
// 创建分片上传
|
||||
// Create multipart upload
|
||||
let mut create_request = s3_client.create_multipart_upload().bucket(bucket).key(object_key);
|
||||
|
||||
create_request = match encryption_type {
|
||||
@@ -526,7 +526,7 @@ async fn test_multipart_encryption_type(
|
||||
let create_multipart_output = create_request.send().await?;
|
||||
let upload_id = create_multipart_output.upload_id().unwrap();
|
||||
|
||||
// 上传分片
|
||||
// Upload parts
|
||||
let mut completed_parts = Vec::new();
|
||||
for part_number in 1..=total_parts {
|
||||
let start = (part_number - 1) * part_size;
|
||||
@@ -541,7 +541,7 @@ async fn test_multipart_encryption_type(
|
||||
.part_number(part_number as i32)
|
||||
.body(aws_sdk_s3::primitives::ByteStream::from(part_data.to_vec()));
|
||||
|
||||
// SSE-C 需要在每个 UploadPart 请求中包含密钥
|
||||
// SSE-C requires the key on each UploadPart request
|
||||
if matches!(encryption_type, EncryptionType::SSEC) {
|
||||
upload_request = upload_request
|
||||
.sse_customer_algorithm("AES256")
|
||||
@@ -558,10 +558,10 @@ async fn test_multipart_encryption_type(
|
||||
.build(),
|
||||
);
|
||||
|
||||
debug!("{:?} 分片 {} 上传完成", encryption_type, part_number);
|
||||
debug!("{:?} part {} uploaded", encryption_type, part_number);
|
||||
}
|
||||
|
||||
// 完成分片上传
|
||||
// Complete the multipart upload
|
||||
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
|
||||
.set_parts(Some(completed_parts))
|
||||
.build();
|
||||
@@ -575,10 +575,10 @@ async fn test_multipart_encryption_type(
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
// 下载并验证
|
||||
// Download and verify
|
||||
let mut get_request = s3_client.get_object().bucket(bucket).key(object_key);
|
||||
|
||||
// SSE-C 需要在 GET 请求中包含密钥
|
||||
// SSE-C requires the key on GET requests
|
||||
if matches!(encryption_type, EncryptionType::SSEC) {
|
||||
get_request = get_request
|
||||
.sse_customer_algorithm("AES256")
|
||||
@@ -588,7 +588,7 @@ async fn test_multipart_encryption_type(
|
||||
|
||||
let get_response = get_request.send().await?;
|
||||
|
||||
// 验证加密头
|
||||
// Verify encryption headers
|
||||
match encryption_type {
|
||||
EncryptionType::SSEKMS => {
|
||||
assert_eq!(
|
||||
@@ -601,11 +601,11 @@ async fn test_multipart_encryption_type(
|
||||
}
|
||||
}
|
||||
|
||||
// 验证数据完整性
|
||||
// Verify data integrity
|
||||
let downloaded_data = get_response.body.collect().await?.into_bytes();
|
||||
assert_eq!(downloaded_data.len(), total_size);
|
||||
assert_eq!(&downloaded_data[..], &test_data[..]);
|
||||
|
||||
info!("✅ {:?} 分片上传测试通过", encryption_type);
|
||||
info!("✅ {:?} multipart upload test passed", encryption_type);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -346,7 +346,7 @@ impl KMSTestSuite {
|
||||
/// Run the complete test suite
|
||||
pub async fn run_test_suite(&self) -> Vec<TestResult> {
|
||||
init_logging();
|
||||
info!("🚀 开始KMS统一测试套件");
|
||||
info!("🚀 Starting unified KMS test suite");
|
||||
|
||||
let start_time = Instant::now();
|
||||
let mut results = Vec::new();
|
||||
@@ -359,17 +359,17 @@ impl KMSTestSuite {
|
||||
.filter(|test| !self.config.include_critical_only || test.is_critical)
|
||||
.collect();
|
||||
|
||||
info!("📊 测试计划: {} 个测试将被执行", tests_to_run.len());
|
||||
info!("📊 Test plan: {} test(s) scheduled", tests_to_run.len());
|
||||
for (i, test) in tests_to_run.iter().enumerate() {
|
||||
info!(" {}. {} ({})", i + 1, test.name, test.category.as_str());
|
||||
}
|
||||
|
||||
// Execute tests
|
||||
for (i, test_def) in tests_to_run.iter().enumerate() {
|
||||
info!("🧪 执行测试 {}/{}: {}", i + 1, tests_to_run.len(), test_def.name);
|
||||
info!(" 📝 描述: {}", test_def.description);
|
||||
info!(" 🏷️ 分类: {}", test_def.category.as_str());
|
||||
info!(" ⏱️ 预计时间: {:?}", test_def.estimated_duration);
|
||||
info!("🧪 Running test {}/{}: {}", i + 1, tests_to_run.len(), test_def.name);
|
||||
info!(" 📝 Description: {}", test_def.description);
|
||||
info!(" 🏷️ Category: {}", test_def.category.as_str());
|
||||
info!(" ⏱️ Estimated duration: {:?}", test_def.estimated_duration);
|
||||
|
||||
let test_start = Instant::now();
|
||||
let result = self.run_single_test(test_def).await;
|
||||
@@ -377,11 +377,11 @@ impl KMSTestSuite {
|
||||
|
||||
match result {
|
||||
Ok(_) => {
|
||||
info!("✅ 测试通过: {} ({:.2}s)", test_def.name, test_duration.as_secs_f64());
|
||||
info!("✅ Test passed: {} ({:.2}s)", test_def.name, test_duration.as_secs_f64());
|
||||
results.push(TestResult::success(test_def.name.clone(), test_def.category.clone(), test_duration));
|
||||
}
|
||||
Err(e) => {
|
||||
error!("❌ 测试失败: {} ({:.2}s): {}", test_def.name, test_duration.as_secs_f64(), e);
|
||||
error!("❌ Test failed: {} ({:.2}s): {}", test_def.name, test_duration.as_secs_f64(), e);
|
||||
results.push(TestResult::failure(
|
||||
test_def.name.clone(),
|
||||
test_def.category.clone(),
|
||||
@@ -393,7 +393,7 @@ impl KMSTestSuite {
|
||||
|
||||
// Add delay between tests to avoid resource conflicts
|
||||
if i < tests_to_run.len() - 1 {
|
||||
debug!("⏸️ 等待2秒后执行下一个测试...");
|
||||
debug!("⏸️ Waiting two seconds before the next test...");
|
||||
sleep(Duration::from_secs(2)).await;
|
||||
}
|
||||
}
|
||||
@@ -408,22 +408,22 @@ impl KMSTestSuite {
|
||||
async fn run_single_test(&self, test_def: &TestDefinition) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
// This is a placeholder for test dispatch logic
|
||||
// In a real implementation, this would dispatch to actual test functions
|
||||
warn!("⚠️ 测试函数 '{}' 在统一运行器中尚未实现,跳过", test_def.name);
|
||||
warn!("⚠️ Test '{}' is not implemented in the unified runner; skipping", test_def.name);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Print comprehensive test summary
|
||||
fn print_test_summary(&self, results: &[TestResult], total_duration: Duration) {
|
||||
info!("📊 KMS测试套件总结");
|
||||
info!("⏱️ 总执行时间: {:.2}秒", total_duration.as_secs_f64());
|
||||
info!("📈 总测试数量: {}", results.len());
|
||||
info!("📊 KMS test suite summary");
|
||||
info!("⏱️ Total duration: {:.2} seconds", total_duration.as_secs_f64());
|
||||
info!("📈 Total tests: {}", results.len());
|
||||
|
||||
let passed = results.iter().filter(|r| r.success).count();
|
||||
let failed = results.iter().filter(|r| !r.success).count();
|
||||
|
||||
info!("✅ 通过: {}", passed);
|
||||
info!("❌ 失败: {}", failed);
|
||||
info!("📊 成功率: {:.1}%", (passed as f64 / results.len() as f64) * 100.0);
|
||||
info!("✅ Passed: {}", passed);
|
||||
info!("❌ Failed: {}", failed);
|
||||
info!("📊 Success rate: {:.1}%", (passed as f64 / results.len() as f64) * 100.0);
|
||||
|
||||
// Summary by category
|
||||
let mut category_summary: std::collections::HashMap<TestCategory, (usize, usize)> = std::collections::HashMap::new();
|
||||
@@ -435,7 +435,7 @@ impl KMSTestSuite {
|
||||
}
|
||||
}
|
||||
|
||||
info!("📊 分类汇总:");
|
||||
info!("📊 Category summary:");
|
||||
for (category, (total, passed_count)) in category_summary {
|
||||
info!(
|
||||
" 🏷️ {}: {}/{} ({:.1}%)",
|
||||
@@ -448,7 +448,7 @@ impl KMSTestSuite {
|
||||
|
||||
// List failed tests
|
||||
if failed > 0 {
|
||||
warn!("❌ 失败的测试:");
|
||||
warn!("❌ Failing tests:");
|
||||
for result in results.iter().filter(|r| !r.success) {
|
||||
warn!(
|
||||
" - {}: {}",
|
||||
@@ -479,7 +479,7 @@ async fn test_kms_critical_suite() -> Result<(), Box<dyn std::error::Error + Sen
|
||||
return Err(format!("Critical test suite failed: {failed_count} tests failed").into());
|
||||
}
|
||||
|
||||
info!("✅ 所有关键测试通过");
|
||||
info!("✅ All critical tests passed");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -494,13 +494,13 @@ async fn test_kms_full_suite() -> Result<(), Box<dyn std::error::Error + Send +
|
||||
let failed_count = results.iter().filter(|r| !r.success).count();
|
||||
let success_rate = ((total_tests - failed_count) as f64 / total_tests as f64) * 100.0;
|
||||
|
||||
info!("📊 完整测试套件结果: {:.1}% 成功率", success_rate);
|
||||
info!("📊 Full suite success rate: {:.1}%", success_rate);
|
||||
|
||||
// Allow up to 10% failure rate for non-critical tests
|
||||
if success_rate < 90.0 {
|
||||
return Err(format!("Test suite success rate too low: {success_rate:.1}%").into());
|
||||
}
|
||||
|
||||
info!("✅ 完整测试套件通过");
|
||||
info!("✅ Full test suite succeeded");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user