gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

feature: support kms && encryt (#573)

Browse Source 
* feat(kms): implement key management service with local and vault backends

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(kms): enhance security with zeroize for sensitive data and improve key management

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove Hashi word

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor: remove unused request structs from kms handlers

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
This commit is contained in:
guojidan
2025-09-22 17:53:05 +08:00
committed by GitHub
parent f7e188eee7
commit 9ddf6a011d
59 changed files with 18461 additions and 830 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+239
View File
@@ -0,0 +1,239 @@
# RustFS 文档中心
欢迎来到 RustFS 分布式文件系统文档中心！
## 📚 文档导航
### 🔐 KMS (密钥管理服务)
RustFS KMS 提供企业级密钥管理和数据加密服务。
| 文档 | 描述 | 适用场景 |
|------|------|----------|
| [KMS 使用指南](./kms/README.md) | 完整的 KMS 使用文档，包含快速开始、配置和部署 | 所有用户必读 |
| [HTTP API 接口](./kms/http-api.md) | HTTP REST API 接口文档和使用示例 | 管理员和运维 |
| [编程 API 接口](./kms/api.md) | Rust 库编程接口和代码示例 | 开发者集成 |
| [配置参考](./kms/configuration.md) | 完整的配置选项和环境变量说明 | 系统管理员 |
| [故障排除](./kms/troubleshooting.md) | 常见问题诊断和解决方案 | 运维人员 |
| [安全指南](./kms/security.md) | 安全最佳实践和合规指导 | 安全架构师 |
## 🚀 快速开始
### 1. KMS 5分钟快速部署
**生产环境（使用 Vault）**
```bash
# 1. 启用 Vault 功能编译
cargo build --features vault --release
# 2. 配置环境变量
export RUSTFS_VAULT_ADDRESS=https://vault.company.com:8200
export RUSTFS_VAULT_TOKEN=hvs.CAESIJ...
# 3. 启动服务
./target/release/rustfs server
```
**开发测试（使用本地后端）**
```bash
# 1. 编译测试版本
cargo build --release
# 2. 配置本地存储
export RUSTFS_KMS_BACKEND=Local
export RUSTFS_KMS_LOCAL_KEY_DIR=/tmp/rustfs-keys
# 3. 启动服务
./target/release/rustfs server
```
### 2. S3 兼容加密
```bash
# 上传加密文件
curl -X PUT https://rustfs.company.com/bucket/sensitive.txt \
-H "x-amz-server-side-encryption: AES256" \
--data-binary @sensitive.txt
# 自动解密下载
curl https://rustfs.company.com/bucket/sensitive.txt
```
## 🏗️ 架构概览
### KMS 三层安全架构
```
┌─────────────────────────────────────────────────┐
│ 应用层 │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ S3 API │ │ REST API │ │
│ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────┤
│ 加密层 │
│ ┌─────────────┐ 加密 ┌─────────────────┐ │
│ │ 对象数据 │ ◄───► │ 数据密钥 (DEK) │ │
│ └─────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────┤
│ 密钥管理层 │
│ ┌─────────────────┐ 加密 ┌──────────────┐ │
│ │ 数据密钥 (DEK) │ ◄────│ 主密钥 │ │
│ └─────────────────┘ │ (Vault/HSM) │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────┘
```
### 核心特性
- ✅ **多层加密**: Master Key → DEK → Object Data
- ✅ **高性能**: 1MB 流式加密，支持大文件
- ✅ **多后端**: Vault (生产) + Local (测试)
- ✅ **S3 兼容**: 支持标准 SSE-S3/SSE-KMS 头
- ✅ **企业级**: 审计、监控、合规支持
## 📖 学习路径
### 👨‍💻 开发者
1. 阅读 [编程 API 接口](./kms/api.md) 了解 Rust 库使用
2. 查看代码示例学习集成方法
3. 参考 [故障排除](./kms/troubleshooting.md) 解决问题
### 👨‍💼 系统管理员
1. 从 [KMS 使用指南](./kms/README.md) 开始
2. 学习 [HTTP API 接口](./kms/http-api.md) 进行管理
3. 详细阅读 [配置参考](./kms/configuration.md)
4. 设置监控和日志
### 👨‍🔧 运维工程师
1. 熟悉 [HTTP API 接口](./kms/http-api.md) 进行日常管理
2. 掌握 [故障排除](./kms/troubleshooting.md) 技能
3. 了解 [安全指南](./kms/security.md) 要求
4. 建立运维流程
### 🔒 安全架构师
1. 深入学习 [安全指南](./kms/security.md)
2. 评估威胁模型和风险
3. 制定安全策略
## 🤝 贡献指南
我们欢迎社区贡献！
### 文档贡献
```bash
# 1. Fork 项目
git clone https://github.com/your-username/rustfs.git
# 2. 创建文档分支
git checkout -b docs/improve-kms-guide
# 3. 编辑文档
# 编辑 docs/kms/ 下的 Markdown 文件
# 4. 提交更改
git add docs/
git commit -m "docs: improve KMS configuration examples"
# 5. 创建 Pull Request
gh pr create --title "Improve KMS documentation"
```
### 文档规范
- 使用清晰的标题和结构
- 提供可运行的代码示例
- 包含适当的警告和提示
- 支持多种使用场景
- 保持内容最新
## 📞 支持与反馈
### 获取帮助
- **GitHub Issues**: https://github.com/rustfs/rustfs/issues
- **讨论区**: https://github.com/rustfs/rustfs/discussions
- **文档问题**: 在相关文档页面创建 Issue
- **安全问题**: security@rustfs.com
### 问题报告模板
报告问题时请提供：
```markdown
**环境信息**
- RustFS 版本: v1.0.0
- 操作系统: Ubuntu 20.04
- Rust 版本: 1.75.0
**问题描述**
简要描述遇到的问题...
**重现步骤**
1. 步骤一
2. 步骤二
3. 步骤三
**期望行为**
描述期望的正确行为...
**实际行为**
描述实际发生的情况...
**相关日志**
```bash
# 粘贴相关日志
```
**附加信息**
其他可能有用的信息...
```
## 📈 版本历史
| 版本 | 发布日期 | 主要特性 |
|------|----------|----------|
| v1.0.0 | 2024-01-15 | 🎉 首个正式版本，完整 KMS 功能 |
| v0.9.0 | 2024-01-01 | 🔐 KMS 系统重构，性能优化 |
| v0.8.0 | 2023-12-15 | ⚡ 流式加密，1MB 块大小优化 |
## 🗺️ 开发路线图
### 即将发布 (v1.1.0)
- [ ] 密钥自动轮转
- [ ] HSM 集成支持
- [ ] Web UI 管理界面
- [ ] 更多合规性支持 (SOC2, HIPAA)
### 长期规划
- [ ] 多租户密钥隔离
- [ ] 密钥导入/导出工具
- [ ] 性能基准测试套件
- [ ] Kubernetes Operator
## 📋 文档反馈
帮助我们改进文档！
**这些文档对您有帮助吗？**
- 👍 很有帮助
- 👌 基本满意
- 👎 需要改进
**改进建议**：
请在 GitHub Issues 中提出具体的改进建议。
---
**最后更新**: 2024-01-15
**文档版本**: v1.0.0
*感谢使用 RustFS！我们致力于为您提供最好的分布式文件系统解决方案。*
docs/kms/README.md
+151
View File
@@ -0,0 +1,151 @@
# RustFS Key Management Service
The RustFS Key Management Service (KMS) provides end-to-end key orchestration, envelope encryption, and S3-compatible semantics for encrypted object storage. It sits between the RustFS API surface and the underlying encryption primitives, ensuring that data at rest and in flight remains protected while keeping operational workflows simple.
## Highlights
- **Multiple backends** – plug in Vault for production or use the Local filesystem backend for development and CI.
- **Envelope encryption** – master keys protect data-encryption keys (DEKs); DEKs protect object payloads with AES-256-GCM streaming.
- **S3 compatibility** – works transparently with `SSE-S3`, `SSE-KMS`, and `SSE-C` headers so existing tools continue to function.
- **Dynamic lifecycle** – configure, rotate, or swap backends at runtime by calling the admin REST API; no server restart is required.
- **Caching & resilience** – built-in caching minimises latency, while health probes, retries, and metrics help operators keep track of the service.
## Architecture
```
┌──────────────────────────────────────────────────────────┐
│ RustFS Frontend │
│ (S3 compatible API, IAM, policy engine, bucket logic) │
└──────────────┬───────────────────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Encryption Service Manager │
│ • Applies admin config │
│ • Controls backend runtime │
│ • Exposes metrics / health │
└──────────────┬──────────────┘
│
┌─────────┴─────────┐
│ │
▼ ▼
┌────────────────┐ ┌────────────────────┐
│ Local Backend │ │ Vault Backend │
│ • File-based │ │ • Transit engine │
│ • Dev / CI │ │ • Production ready │
└────────────────┘ └────────────────────┘
```
### Components at a Glance
| Component | Responsibility |
|------------------------------|-------------------------------------------------------------------------|
| `rustfs::kms::manager` | Owns backend lifecycle, caching, and key orchestration. |
| `rustfs::kms::encryption` | Encrypts/decrypts payloads, issues data keys, validates headers. |
| Admin REST handlers | Accept configuration requests (`configure`, `start`, `status`, etc.). |
| Backends | `local` (filesystem) and `vault` (Transit) implementations. |
## Supported Backends
| Backend | When to use | Key storage | Authentication | Notes |
|---------|-------------|-------------|----------------|-------|
| Local | Development, CI, integration tests | JSON-encoded key blobs on disk | none | Simple, fast to bootstrap, not secure for production. |
| Vault | Production or pre-production | Vault Transit & KV engines | token or AppRole | Supports rotation, audit logging, sealed-state recovery, TLS. |
Refer to [configuration.md](configuration.md) for static configuration details and [dynamic-configuration-guide.md](dynamic-configuration-guide.md) for the runtime workflow.
## Encryption Workflows
RustFS KMS supports the same S3 semantics users expect:
- **SSE-S3** – RustFS manages the data key lifecycle and returns the `x-amz-server-side-encryption` header.
- **SSE-KMS** – RustFS issues per-object data keys bound to the configured KMS backend, exposing the `x-amz-server-side-encryption` header with value `aws:kms`.
- **SSE-C** – Clients provide a 256-bit key and MD5 checksum per request; RustFS uses KMS to encrypt metadata, while encrypted payloads are streamed with the customer key.
Internally, every object follows the envelope-encryption flow below:
1. Determine the logical key-id (default, explicit header, or SSE-C customer key).
2. Ask the configured backend for a DEK or encryption context.
3. Stream-encrypt the payload with AES-256-GCM (1 MiB chunking, authenticated headers).
4. Persist metadata (IV, checksum, key-id) alongside object state.
5. During GET/HEAD, the same process runs in reverse with integrity checks.
## Quick Start
1. **Build RustFS** – `cargo build --release` or run the project-specific build helper.
2. **Prepare credentials** – ensure you have admin access keys; for Vault, export `VAULT_ADDR` and a root or scoped token.
3. **Launch RustFS** – `./target/release/rustfs server` (KMS starts in `NotConfigured`).
4. **Configure the backend**:
```bash
# Local backend (ephemeral testing)
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "local",
"key_dir": "/var/lib/rustfs/kms-keys",
"default_key_id": "rustfs-master"
}' \
http://localhost:9000/rustfs/admin/v3/kms/configure
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/start
```
```bash
# Vault backend (production)
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": {
"token": "s.XYZ..."
},
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master"
}' \
https://rustfs.example.com/rustfs/admin/v3/kms/configure
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST https://rustfs.example.com/rustfs/admin/v3/kms/start
```
5. **Verify**:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
The response should include `"status": "Running"` and the configured backend summary.
## Documentation Map
| Topic | Description |
|-------|-------------|
| [http-api.md](http-api.md) | Formal REST endpoint reference with request/response samples. |
| [dynamic-configuration-guide.md](dynamic-configuration-guide.md) | Gradual rollout, rotation, and failover playbooks. |
| [configuration.md](configuration.md) | Static configuration files, environment variables, Helm/Ansible hints. |
| [api.md](api.md) | Rust crate interfaces (`ConfigureKmsRequest`, `KmsManager`, encryption helpers). |
| [sse-integration.md](sse-integration.md) | Mapping between S3 headers and RustFS behaviour, client examples. |
| [security.md](security.md) | Threat model, access control, TLS, auditing, secrets hygiene. |
| [test_suite_integration.md](test_suite_integration.md) | Running e2e, Vault, and regression test suites. |
| [troubleshooting.md](troubleshooting.md) | Common errors and recovery steps. |
## Terminology
| Term | Definition |
|------|------------|
| **KMS backend** | Implementation that holds master keys (Local filesystem or Vault Transit/ KV). |
| **Master Key** | Root key stored in the backend; encrypts data keys. |
| **Data Encryption Key (DEK)** | Per-object key that encrypts payload chunks. |
| **Envelope Encryption** | Wrapping DEKs with a higher-level key before persisting. |
| **SSE-S3 / SSE-KMS / SSE-C** | Amazon S3-compatible encryption modes supported by RustFS. |
For deeper dives continue with the documents referenced above. EOF
docs/kms/api.md
+169
View File
@@ -0,0 +1,169 @@
# RustFS KMS Developer API
This document targets developers extending RustFS or embedding the KMS primitives directly. The `rustfs-kms` crate exposes building blocks for configuration, backend orchestration, and data-key lifecycle management.
## Crate Overview
Add the crate to your workspace (already included in RustFS):
```toml
[dependencies]
rustfs-kms = { path = "crates/kms" }
```
Key namespaces:
| Module | Purpose |
|--------|---------|
| `rustfs_kms::config` | Typed configuration objects for local/vault backends. |
| `rustfs_kms::manager::KmsManager` | High-level coordinator that proxies operations to a backend. |
| `rustfs_kms::encryption::service::EncryptionService` | Frontend consumed by RustFS S3 handlers. |
| `rustfs_kms::backends` | Backend trait definitions and concrete implementations. |
| `rustfs_kms::types` | Request/response DTOs used by the REST handlers and manager. |
| `rustfs_kms::service_manager` | Async runtime that powers `/kms/configure`, `/kms/start`, etc. |
## Constructing a Configuration
```rust
use rustfs_kms::config::{BackendConfig, KmsBackend, KmsConfig, LocalConfig, VaultConfig, VaultAuthMethod};
let config = KmsConfig {
backend: KmsBackend::Vault,
backend_config: BackendConfig::Vault(VaultConfig {
address: "https://vault.example.com:8200".parse().unwrap(),
auth_method: VaultAuthMethod::Token { token: "s.XYZ".into() },
namespace: None,
mount_path: "transit".into(),
kv_mount: "secret".into(),
key_path_prefix: "rustfs/kms/keys".into(),
tls: None,
}),
default_key_id: Some("rustfs-master".into()),
timeout: std::time::Duration::from_secs(30),
retry_attempts: 3,
enable_cache: true,
cache_config: Default::default(),
};
```
To build configurations from the admin REST payloads, use `api_types::ConfigureKmsRequest`:
```rust
use rustfs_kms::api_types::{ConfigureKmsRequest, ConfigureVaultKmsRequest};
let request = ConfigureKmsRequest::Vault(ConfigureVaultKmsRequest {
address: "https://vault.example.com:8200".into(),
auth_method: VaultAuthMethod::Token { token: "s.XYZ".into() },
namespace: None,
mount_path: Some("transit".into()),
kv_mount: Some("secret".into()),
key_path_prefix: Some("rustfs/kms/keys".into()),
default_key_id: Some("rustfs-master".into()),
skip_tls_verify: Some(false),
timeout_seconds: Some(30),
retry_attempts: Some(5),
enable_cache: Some(true),
max_cached_keys: Some(2048),
cache_ttl_seconds: Some(600),
});
let kms_config: KmsConfig = (&request).into();
```
## Service Manager Lifecycle
The admin layer interacts with a `ServiceManager` singleton that wraps `KmsManager`:
```rust
use rustfs_kms::{init_global_kms_service_manager, get_global_kms_service_manager};
let manager = init_global_kms_service_manager();
manager.configure(config).await?;
manager.start().await?;
let status = manager.get_status().await; // -> KmsServiceStatus::Running
```
`get_global_encryption_service()` returns the `EncryptionService` façade that the S3 request handlers call. The service exposes async methods mirroring AWS KMS semantics:
```rust
use rustfs_kms::types::{CreateKeyRequest, KeyUsage, GenerateDataKeyRequest, KeySpec};
use rustfs_kms::get_global_encryption_service;
let service = get_global_encryption_service().await.expect("service not initialised");
let create = CreateKeyRequest {
key_name: None,
key_usage: KeyUsage::EncryptDecrypt,
description: Some("project-alpha".into()),
tags: Default::default(),
origin: None,
policy: None,
};
let created = service.create_key(create).await?;
let data_key = service
.generate_data_key(GenerateDataKeyRequest {
key_id: created.key_id.clone(),
key_spec: KeySpec::Aes256,
encryption_context: Default::default(),
})
.await?;
```
## Backend Integration Points
To add a custom backend:
1. Implement the `KmsBackend` trait (see `crates/kms/src/backends/mod.rs`).
2. Provide conversions from `ConfigureKmsRequest` into your backend’s config struct.
3. Register the backend in `BackendFactory` and extend the admin handlers to accept the new `backend_type` string.
The trait contract requires implementing methods such as `create_key`, `encrypt`, `decrypt`, `generate_data_key`, `list_keys`, and `health_check`.
```rust
#[async_trait::async_trait]
pub trait KmsBackend: Send + Sync {
async fn create_key(&self, request: CreateKeyRequest) -> Result<CreateKeyResponse>;
async fn encrypt(&self, request: EncryptRequest) -> Result<EncryptResponse>;
async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse>;
async fn generate_data_key(&self, request: GenerateDataKeyRequest) -> Result<GenerateDataKeyResponse>;
async fn describe_key(&self, request: DescribeKeyRequest) -> Result<DescribeKeyResponse>;
async fn list_keys(&self, request: ListKeysRequest) -> Result<ListKeysResponse>;
async fn delete_key(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse>;
async fn cancel_key_deletion(&self, request: CancelKeyDeletionRequest) -> Result<CancelKeyDeletionResponse>;
async fn health_check(&self) -> Result<bool>;
}
```
## Encryption Pipeline Helpers
`EncryptionService` contains two methods used by the S3 PUT/GET pipeline:
- `encrypt_stream` (invoked by `PutObject` and multipart uploads) obtains DEKs, encrypts payload chunks with AES-256-GCM, and returns headers.
- `decrypt_stream` resolves metadata, fetches the required DEK or customer key, and streams plaintext back to the client.
Both rely on `ObjectCipher` implementations defined in `crates/kms/src/encryption/ciphers.rs`. When adjusting chunk sizes or cipher suites, update these implementations and the SSE documentation.
## Testing Utilities
- `rustfs_kms::mock` contains in-memory backends used by unit tests.
- The e2e crate (`crates/e2e_test`) exposes helpers such as `LocalKMSTestEnvironment` and `VaultTestEnvironment` for integration testing.
- Run the full suite: `cargo test --workspace --exclude e2e_test` for unit coverage, `cargo test -p e2e_test kms:: -- --nocapture` for end-to-end validation.
## Error Handling Conventions
All public async methods return `rustfs_kms::error::Result<T>`. Errors are categorised as:
| Variant | Meaning |
|---------|---------|
| `KmsError::Configuration` | Invalid or missing backend configuration. |
| `KmsError::Backend` | Underlying backend failure (Vault error, disk I/O, etc.). |
| `KmsError::Crypto` | Integrity or cryptographic failure. |
| `KmsError::Cache` | Cache lookup or eviction failure. |
Map these errors to HTTP responses using the helper macros in `rustfs/src/admin/handlers`.
---
For operational workflows continue with [http-api.md](http-api.md) and [dynamic-configuration-guide.md](dynamic-configuration-guide.md). For encryption semantics, see [sse-integration.md](sse-integration.md).
docs/kms/configuration.md
+125
View File
@@ -0,0 +1,125 @@
# KMS Configuration Guide
This guide describes the configuration surfaces for the RustFS Key Management Service. RustFS can be configured statically at process start or dynamically via the admin REST API. Most operators start with a static bootstrap (CLI flags, configuration file, or environment variables) and then rely on dynamic configuration to rotate keys or swap backends.
## Configuration Sources
| Mechanism | When to use | Notes |
|---------------------|----------------------------------------------------------|-------|
| CLI flags | Local development, ad-hoc testing | `rustfs server --kms-enable --kms-backend vault ...` |
| Environment vars | Container/Helm/Ansible deployments | Prefix variables with `RUSTFS_` (see table below). |
| Static config file | Use your orchestration tooling to render TOML/YAML, then pass the corresponding flags during startup. |
| Dynamic REST API | Post-start updates without restarting (see [dynamic-configuration-guide.md](dynamic-configuration-guide.md)). |
## CLI Flags & Environment Variables
| CLI flag | Env variable | Description |
|-----------------------------|--------------------------------|-------------|
| `--kms-enable` | `RUSTFS_KMS_ENABLE` | Enables KMS at startup. Defaults to `false`. |
| `--kms-backend <local|vault>` | `RUSTFS_KMS_BACKEND` | Selects the backend implementation. Defaults to `local`. |
| `--kms-key-dir <path>` | `RUSTFS_KMS_KEY_DIR` | Required when `kms-backend=local`; directory that stores wrapped master keys. |
| `--kms-vault-address <url>` | `RUSTFS_KMS_VAULT_ADDRESS` | Vault base URL (e.g. `https://vault.example.com:8200`). |
| `--kms-vault-token <token>` | `RUSTFS_KMS_VAULT_TOKEN` | Token used for Vault authentication. Prefer AppRole or short-lived tokens. |
| `--kms-default-key-id <id>` | `RUSTFS_KMS_DEFAULT_KEY_ID` | Default key used when clients omit `x-amz-server-side-encryption-aws-kms-key-id`. |
> **Tip:** Even when you plan to reconfigure the backend dynamically, setting `--kms-enable` is useful because it instantiates the global manager eagerly and surfaces better error messages when configuration fails.
## Static TOML Example (Local Backend)
```toml
# rustfs.toml
[kms]
enabled = true
backend = "local"
key_dir = "/var/lib/rustfs/kms-keys"
default_key_id = "rustfs-master"
```
Render this file using your favourite template tool and translate it to CLI flags when launching RustFS:
```bash
rustfs server \
--kms-enable \
--kms-backend local \
--kms-key-dir /var/lib/rustfs/kms-keys \
--kms-default-key-id rustfs-master
```
## Static TOML Example (Vault Backend)
```toml
[kms]
enabled = true
backend = "vault"
vault_address = "https://vault.example.com:8200"
# Supply either a token or render AppRole credentials dynamically
vault_token = "s.XYZ..."
default_key_id = "rustfs-master"
```
Ensure that the Vault binary is reachable and the Transit engine is initialised before starting RustFS:
```bash
vault secrets enable transit
vault secrets enable -path=secret kv-v2
vault write transit/keys/rustfs-master type=aes256-gcm96
```
If you prefer AppRole authentication, omit `vault_token` and set the token dynamically via the REST API once RustFS is online (see [dynamic-configuration-guide.md](dynamic-configuration-guide.md)).
## Backend-Specific Options
### Local Backend
| Field | Description |
|------------------|-------------|
| `key_dir` | Directory where wrapped master keys are stored (`*.key` JSON files). Ensure it is backed up securely in persistent deployments. |
| `default_key_id` | Optional; if not provided, SSE-S3 uploads require an explicit header. |
| `file_permissions` (REST only) | Octal permissions applied to generated key files (`0o600` by default). |
| `master_key` (REST only) | Base64-encoded wrapping key used to protect DEKs on disk. Leave unset to generate one automatically. |
During development you can generate a default key manually:
```bash
mkdir -p /tmp/rustfs-keys
openssl rand -hex 32 > /tmp/rustfs-keys/rustfs-master.material
```
The KMS e2e tests also demonstrate programmatic key creation using the `/kms/keys` API.
### Vault Backend
| Field | Description |
|---------------------|-------------|
| `address` | Base URL including scheme. TLS is strongly recommended. |
| `auth_method` | `Token { token: "..." }` or `AppRole { role_id, secret_id }`. Tokens should be renewable or short-lived. |
| `mount_path` | Transit engine mount (default `transit`). |
| `kv_mount` | KV v2 engine used to stash wrapped keys or metadata. |
| `key_path_prefix` | Prefix under the KV mount (e.g. `rustfs/kms/keys`). |
| `namespace` | Vault enterprise namespace (optional). |
| `skip_tls_verify` | Development convenience; avoid using this in production. |
| `default_key_id` | Transit key to use when clients omit `x-amz-server-side-encryption-aws-kms-key-id`. |
## Advanced Runtime Knobs (REST API)
The dynamic API exposes additional fields not available on the CLI:
| Field | Purpose |
|-------|---------|
| `timeout_seconds` | Backend operation timeout (defaults to 30s). |
| `retry_attempts` | Number of retries for transient backend failures (defaults to 3). |
| `enable_cache` | Enables in-memory cache of DEKs and metadata. |
| `max_cached_keys` / `cache_ttl_seconds` | Cache size and TTL limits. |
These options are mostly relevant for large deployments; configure them via the `/kms/configure` REST call once the service is online.
## Bootstrapping Workflow
1. Pick a backend (`local` or `vault`).
2. Ensure the required infrastructure is ready (filesystem permissions or Vault engines).
3. Start RustFS with `--kms-enable` and the minimal bootstrap flags.
4. Call the REST API to refine configuration (timeouts, cache, AppRole, etc.).
5. Verify with `/kms/status` and issue a test `PutObject` using SSE headers.
6. Record the configuration in your infra-as-code tooling for repeatability.
For runtime reconfiguration (rotating keys, swapping from local to Vault) follow the step-by-step guide in [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/dynamic-configuration-guide.md
+155
View File
@@ -0,0 +1,155 @@
# Dynamic KMS Configuration Playbook
RustFS exposes a first-class admin REST API that allows you to configure, start, stop, and reconfigure the KMS subsystem without restarting the server. This document walks through common operational scenarios.
## Prerequisites
- RustFS is running and reachable on the admin endpoint (typically `http(s)://<host>/rustfs/admin/v3`).
- You have admin access and credentials (access key/secret or session token) with the `ServerInfoAdminAction` permission.
- Optional: `awscurl` or another SigV4-aware HTTP client to sign admin requests.
Before starting, confirm the KMS service manager is initialised:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
The initial response shows `"status": "NotConfigured"`.
## Initial Configuration Flow
1. **Submit the configuration**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "local",
"key_dir": "/var/lib/rustfs/kms-keys",
"default_key_id": "rustfs-master",
"enable_cache": true,
"cache_ttl_seconds": 900
}' \
http://localhost:9000/rustfs/admin/v3/kms/configure
```
2. **Start the service**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/start
```
3. **Verify**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
Look for `"status": "Running"` and a backend summary.
## Switching to Vault
To migrate from the local backend to Vault:
1. Prepare Vault:
```bash
vault secrets enable transit
vault secrets enable -path=secret kv-v2
vault write transit/keys/rustfs-master type=aes256-gcm96
```
2. Configure the new backend without stopping service:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": { "approle": { "role_id": "...", "secret_id": "..." } },
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master",
"retry_attempts": 5,
"timeout_seconds": 60
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
3. Confirm the new backend is active via `/kms/status`.
4. Run test uploads with `SSE-KMS` headers to ensure the new backend is serving requests.
## Rotating the Default Key
1. **Create a new key** using the key management API:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{ "KeyUsage": "ENCRYPT_DECRYPT", "Description": "rotation-2024-09" }' \
http://localhost:9000/rustfs/admin/v3/kms/keys
```
2. **Set it as default** via `reconfigure`:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"backend_type": "vault",
"default_key_id": "rotation-2024-09"
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
Only the fields supplied in the payload are updated; omitted fields keep their previous values.
3. **Validate** by uploading a new object and checking that `x-amz-server-side-encryption-aws-kms-key-id` reports the new key.
## Rolling Cache or Timeout Changes
Caching knobs help tune latency. To adjust them at runtime:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST -d '{
"enable_cache": true,
"max_cached_keys": 2048,
"cache_ttl_seconds": 600,
"timeout_seconds": 20
}' \
http://localhost:9000/rustfs/admin/v3/kms/reconfigure
```
## Pausing the KMS Service
Stopping the service keeps configuration in place but disables new KMS operations. Existing SSE objects remain accessible only if their metadata allows offline decryption.
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/stop
```
Restart later with `/kms/start`.
## Automation Tips
- Wrap REST calls in an idempotent script (see `scripts/` for examples) so you can re-run configuration safely.
- Use `--test-threads=1` when running KMS e2e suites in CI; they spin up real servers and Vault instances.
- In Kubernetes, run the configuration script as an init job that waits for both RustFS and Vault readiness before calling `/kms/configure`.
- Emit events to your observability platform: successful reconfigurations generate structured logs with the backend summary.
## Rollback Strategy
If a new configuration introduces errors:
1. Call `/kms/reconfigure` with the previous payload (keep a snapshot in version control).
2. If the backend is unreachable, call `/kms/stop` to protect data from partial writes.
3. Investigate logs under `rustfs::kms::*` and Vault audit logs.
4. Once the issue is resolved, reapply the desired configuration and restart.
Dynamic configuration makes backend maintenance safe and repeatable—ensure every change is scripted and traceable.
docs/kms/frontend-api-guide-zh.md
+2382
View File
File diff suppressed because it is too large Load Diff
docs/kms/http-api.md
+248
View File
@@ -0,0 +1,248 @@
# KMS Admin HTTP API Reference
The RustFS KMS admin API is exposed under the admin prefix (`/rustfs/admin/v3`). Requests must be signed with SigV4 credentials that have the `ServerInfoAdminAction` permission. All request and response bodies use JSON, and all endpoints return standard HTTP status codes.
- Base URL examples: `http://localhost:9000/rustfs/admin/v3`, `https://rustfs.example.com/rustfs/admin/v3`.
- Headers: set `Content-Type: application/json` for requests with bodies.
- Authentication: sign with SigV4 (`awscurl`, `aws-signature-v4`, or the official SDKs).
## Service Lifecycle
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/kms/configure` | POST | Apply the initial backend configuration. Does not start the service. |
| `/kms/reconfigure` | POST | Merge a new configuration on top of the existing one. |
| `/kms/start` | POST | Start the configured backend. |
| `/kms/stop` | POST | Stop the backend; configuration is kept. |
| `/kms/status` | GET | Lightweight status summary (`Running`, `Configured`, etc.). |
| `/kms/service-status` | GET | Backward-compatible alias for `/kms/status`. |
| `/kms/config` | GET | Returns the cached configuration summary. |
| `/kms/clear-cache` | POST | Clears in-memory DEK and metadata caches. |
### Configure / Reconfigure
**Request**
```json
{
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method": { "token": "s.XYZ" },
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"default_key_id": "rustfs-master",
"enable_cache": true,
"cache_ttl_seconds": 600,
"timeout_seconds": 30,
"retry_attempts": 3
}
```
**Response**
```json
{
"success": true,
"message": "KMS configured successfully",
"status": "Configured"
}
```
> **Partial updates:** `/kms/reconfigure` updates only the fields present in the payload. Use this to rotate tokens or adjust cache parameters without resubmitting the full configuration.
### Start / Stop
**Start response**
```json
{
"success": true,
"message": "KMS service started successfully",
"status": "Running"
}
```
**Stop response**
```json
{
"success": true,
"message": "KMS service stopped successfully",
"status": "Configured"
}
```
### Status & Config
`GET /kms/status`
```json
{
"status": "Running",
"backend_type": "vault",
"healthy": true,
"config_summary": {
"backend_type": "vault",
"default_key_id": "rustfs-master",
"timeout_seconds": 30,
"retry_attempts": 3,
"enable_cache": true,
"cache_summary": {
"max_keys": 1024,
"ttl_seconds": 600,
"enable_metrics": true
},
"backend_summary": {
"backend_type": "vault",
"address": "https://vault.example.com:8200",
"auth_method_type": "token",
"namespace": null,
"mount_path": "transit",
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys"
}
}
}
```
`GET /kms/config`
```json
{
"backend": "vault",
"cache_enabled": true,
"cache_max_keys": 1024,
"cache_ttl_seconds": 600,
"default_key_id": "rustfs-master"
}
```
`POST /kms/clear-cache` returns HTTP `204` with an empty body when successful.
## Key Management
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/kms/keys` | POST | Create a new master key in the backend. |
| `/kms/keys` | GET | List master keys (paginated). |
| `/kms/keys/{key_id}` | GET | Retrieve metadata for a specific key. |
| `/kms/keys/delete` | DELETE | Schedule key deletion. |
| `/kms/keys/cancel-deletion` | POST | Cancel a pending deletion request. |
### Create Key
**Request**
```json
{
"KeyUsage": "ENCRYPT_DECRYPT",
"Description": "project-alpha",
"Tags": {
"owner": "security",
"env": "prod"
}
}
```
**Response**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"key_metadata": {
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"description": "project-alpha",
"enabled": true,
"key_usage": "ENCRYPT_DECRYPT",
"creation_date": "2024-09-18T07:10:42.012345Z",
"rotation_enabled": false
}
}
```
### List Keys
`GET /kms/keys?limit=50&marker=<token>`
```json
{
"keys": [
{ "key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85", "description": "project-alpha" }
],
"truncated": false,
"next_marker": null
}
```
### Describe Key
`GET /kms/keys/fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85`
```json
{
"key_metadata": {
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"description": "project-alpha",
"enabled": true,
"key_usage": "ENCRYPT_DECRYPT",
"creation_date": "2024-09-18T07:10:42.012345Z",
"deletion_date": null
}
}
```
### Delete & Cancel Deletion
**Delete request**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"pending_window_in_days": 7
}
```
**Cancel deletion**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85"
}
```
Both endpoints respond with the updated `key_metadata`.
## Data Key Operations
`POST /kms/generate-data-key`
**Request**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"key_spec": "AES_256",
"encryption_context": {
"bucket": "analytics-data",
"object": "2024/09/18/report.parquet"
}
}
```
**Response**
```json
{
"key_id": "fa5bac0e-2a2c-4f9a-a09d-2f5b8a59ed85",
"plaintext_key": "sQW6qt0yS7CqD6c8hY7GZg==",
"ciphertext_blob": "gAAAAABlLK..."
}
```
- `plaintext_key` is Base64-encoded and must be zeroised after use.
- `ciphertext_blob` can be stored alongside object metadata for future re-wraps.
## Error Handling
| Code | Meaning | Example payload |
|------|---------|-----------------|
| `400 Bad Request` | Malformed JSON or missing required fields. | `{ "code": "InvalidRequest", "message": "invalid JSON" }` |
| `401 Unauthorized` | Request was not signed or credentials are invalid. | `{ "code": "AccessDenied", "message": "authentication required" }` |
| `403 Forbidden` | Caller lacks admin permissions. | `{ "code": "AccessDenied", "message": "unauthorised" }` |
| `409 Conflict` | Backend already configured in an incompatible way. | `{ "code": "Conflict", "message": "KMS already running" }` |
| `500 Internal Server Error` | Backend failure or transient issue. Logs include details. | `{ "code": "InternalError", "message": "failed to create key: ..." }` |
## Useful Utilities
- [`awscurl`](https://github.com/okigan/awscurl) for quick SigV4 requests.
- The `scripts/` directory contains example shell scripts to configure local and Vault backends automatically.
- The e2e test harness (`cargo test -p e2e_test kms:: -- --nocapture`) demonstrates end-to-end API usage against both backends.
For dynamic workflows and automation strategies, continue with [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/security.md
+65
View File
@@ -0,0 +1,65 @@
# KMS Security Guidelines
This document summarises the security posture of the RustFS KMS subsystem and offers guidance for safe production deployment.
## Threat Model
- Attackers might obtain network access to RustFS or Vault.
- Leaked admin credentials could manipulate KMS configuration.
- Misconfigured SSE-C clients could expose plaintext keys.
- Insider threats may attempt to extract master keys from disk-based storage.
RustFS mitigates these risks via access control, auditability, and best practices outlined below.
## Authentication & Authorisation
- The admin API requires SigV4 credentials with `ServerInfoAdminAction`. Restrict these credentials to trusted automation.
- Do **not** share admin credentials with regular S3 clients. Provision separate IAM users for data-plane traffic.
- When running behind a reverse proxy, ensure the proxy passes through headers required for SigV4 signature validation.
## Network Security
- Enforce TLS for both RustFS and Vault deployments. Set `skip_tls_verify=false` in production.
- Use mTLS or private network peering between RustFS and Vault where possible.
- Restrict Vault transit endpoints using network ACLs or service meshes so only RustFS can reach them.
## Secret Management
- Never store Vault tokens directly in configuration files. Prefer AppRole or short-lived tokens injected at runtime.
- If you must render a token (e.g. in CI), use environment variables with limited scope and rotate them frequently.
- For the local backend, keep the key directory on encrypted disks with tight POSIX permissions (default `0o600`).
## Vault Hardening Checklist
- Enable audit logging (`vault audit enable file file_path=/var/log/vault_audit.log`).
- Create a dedicated policy granting access only to the `transit` and `secret` paths used by RustFS.
- Configure automatic token renewal or rely on `vault agent` to manage token lifetimes.
- Monitor the health endpoint (`/v1/sys/health`) and integrate it into your on-call alerts.
## Caching & Memory Hygiene
- When `enable_cache=true`, DEKs are stored in memory for the configured TTL. Tune `max_cached_keys` and TTL to balance latency versus exposure.
- The encryption service zeroises plaintext keys after use. Avoid logging plaintext keys or contexts in custom code.
- For workloads that require strict FIPS compliance, disable caching and rely on Vault for each request.
## SSE-C Considerations
- Clients are responsible for providing 256-bit keys and MD5 hashes. Reject uploads where the digest does not match.
- Educate clients that SSE-C keys are never stored server side; losing the key means losing access to the object.
- Use HTTPS for all client connections to prevent key disclosure.
## Audit & Monitoring
- Capture structured logs emitted under the `rustfs::kms` target. Each admin call logs request principals and outcomes.
- Export metrics such as cache hit ratio, backend latency, and failure counts to your observability stack.
- Periodically run the e2e Vault suite in a staging environment to verify backup/restore procedures.
## Incident Response
1. Stop the KMS service (`POST /kms/stop`) to freeze new operations.
2. Rotate admin credentials and Vault tokens.
3. Examine audit logs to determine the blast radius.
4. Restore keys from backups or Vault versions if tampering occurred.
5. Reconfigure the backend using trusted credentials and restart the service.
By adhering to these practices, you can deploy RustFS KMS with confidence across regulated or high-security environments.
docs/kms/sse-integration.md
+91
View File
@@ -0,0 +1,91 @@
# Server-Side Encryption Integration
RustFS implements Amazon S3-compatible server-side encryption semantics. This document outlines how each mode maps to KMS operations and how clients should format requests.
## Supported Modes
| Mode | Request Headers | Managed by | Notes |
|------|-----------------|------------|-------|
| `SSE-S3` | `x-amz-server-side-encryption: AES256` | RustFS KMS using the configured default key. | Simplest option; clients do not manage keys. |
| `SSE-KMS` | `x-amz-server-side-encryption: aws:kms`<br>`x-amz-server-side-encryption-aws-kms-key-id: <key-id>` (optional) | RustFS KMS + backend (Vault/Local). | Specify a key-id to override the default. |
| `SSE-C` | `x-amz-server-side-encryption-customer-algorithm: AES256`<br>`x-amz-server-side-encryption-customer-key: <Base64 key>`<br>`x-amz-server-side-encryption-customer-key-MD5: <Base64 MD5>` | Customer provided | RustFS never stores the plaintext key; clients must supply it on every request. |
## Request Examples
### SSE-S3 Upload & Download
```bash
# Upload
aws s3api put-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key obj.txt --body file.txt \
--server-side-encryption AES256
# Download
aws s3api get-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key obj.txt out.txt
```
### SSE-KMS with Explicit Key ID
```bash
aws s3api put-object \
--endpoint-url http://localhost:9000 \
--bucket demo --key report.csv --body report.csv \
--server-side-encryption aws:kms \
--ssekms-key-id rotation-2024-09
```
If `--ssekms-key-id` is omitted, RustFS uses the configured `default_key_id`.
### SSE-C Multipart Upload
SSE-C requires additional care:
1. Generate a 256-bit key and compute its MD5 digest.
2. For multipart uploads, every request (initiate, upload-part, complete, GET) must include the SSE-C headers.
3. Keep part sizes ≥ 5 MiB to avoid falling back to inline storage which complicates key handling.
```bash
KEY="01234567890123456789012345678901"
KEY_B64=$(echo -n "$KEY" | base64)
KEY_MD5=$(echo -n "$KEY" | md5 | awk '{print $1}')
aws s3api create-multipart-upload \
--endpoint-url http://localhost:9000 \
--bucket demo --key video.mp4 \
--server-side-encryption-customer-algorithm AES256 \
--server-side-encryption-customer-key "$KEY_B64" \
--server-side-encryption-customer-key-MD5 "$KEY_MD5"
# Upload all parts with the same trio of headers
```
On download, supply the same headers; otherwise the request fails with `AccessDenied`.
## Response Headers
| Header | SSE-S3 | SSE-KMS | SSE-C |
|--------|--------|---------|-------|
| `x-amz-server-side-encryption` | `AES256` | `aws:kms` | _absent_ |
| `x-amz-server-side-encryption-aws-kms-key-id` | _default key id_ | Provided key id | _absent_ |
| `x-amz-server-side-encryption-customer-algorithm` | _absent_ | _absent_ | `AES256` |
| `x-amz-server-side-encryption-customer-key-MD5` | _absent_ | _absent_ | MD5 of supplied key |
## Error Scenarios
| Scenario | Error | Resolution |
|----------|-------|------------|
| SSE-C key/MD5 mismatch | `AccessDenied` | Regenerate the MD5 digest, ensure Base64 encoding is correct. |
| Missing SSE-C headers on GET | `InvalidRequest` | Provide the same `sse-c` headers used during upload. |
| Invalid key id for SSE-KMS | `NotFound` | Call `GET /kms/keys` to retrieve the valid IDs or create one via the admin API. |
| KMS backend offline | `InternalError` | Check `/kms/status`, restart or reconfigure the backend. |
## Best Practices
- Always use HTTPS endpoints when supplying SSE-C headers.
- Log the key-id used for SSE-KMS uploads to simplify forensic analysis.
- For compliance workloads, disable cache or lower cache TTL via `/kms/reconfigure` so data keys are short-lived.
- Test multipart SSE-C flows regularly; the e2e suite (`test_comprehensive_kms_full_workflow`) covers this scenario.
For the administrative API and configuration specifics, refer to [http-api.md](http-api.md) and [configuration.md](configuration.md).
docs/kms/test_suite_integration.md
+77
View File
@@ -0,0 +1,77 @@
# KMS Test Suite Integration
RustFS ships with an extensive set of automated tests that exercise the KMS stack. This guide explains how to run them locally and in CI.
## Crate Overview
- `crates/kms` – unit tests for configuration, caching, and backend adapters.
- `crates/e2e_test/src/kms` – end-to-end suites for Local and Vault backends, multipart uploads, edge cases, and fault recovery.
- `crates/e2e_test/src/kms/common.rs` – reusable test environments (spins up RustFS, configures Vault, manages buckets).
## Prerequisites
| Requirement | Purpose |
|-------------|---------|
| `vault` binary (>=1.15) | Required for Vault end-to-end tests. Install from Vault releases. |
| `awscurl` (optional) | Debugging helper to hit admin endpoints. |
| `openssl`, `md5` | Used by SSE-C helpers during tests. |
| Local ports | Tests bind ephemeral ports (ensure `127.0.0.1:<random>` is free). |
## Running Unit Tests
```bash
cargo test --workspace --exclude e2e_test
```
This covers the core KMS crate plus supporting libraries.
## Running End-to-End Suites
### All KMS Tests
```bash
NO_PROXY=127.0.0.1,localhost \
HTTP_PROXY= HTTPS_PROXY= \
cargo test -p e2e_test kms:: -- --nocapture --test-threads=1
```
- `--nocapture` streams logs to stdout for troubleshooting.
- `--test-threads=1` ensures serial execution; most tests spawn standalone RustFS and Vault processes.
### Local Backend Only
```bash
cargo test -p e2e_test kms::kms_local_test:: -- --nocapture --test-threads=1
```
### Vault Backend Only
```bash
vault server -dev -dev-root-token-id=dev-root-token &
VAULT_PID=$!
cargo test -p e2e_test kms::kms_vault_test:: -- --nocapture --test-threads=1
kill $VAULT_PID
```
The tests can also start Vault automatically if the binary is found on `PATH`. When running in CI, whitelist the `vault` executable in the sandbox or mark the job as privileged.
## Updating Fixtures
- Adjustment to SSE behaviour or multipart limits often requires touching `crates/e2e_test/src/kms/common.rs`. Keep helpers generic so multiple tests can reuse them.
- When fixing bugs, add targeted coverage in the relevant suite (e.g. `kms_fault_recovery_test.rs`).
- Vault-specific fixtures live in `crates/e2e_test/src/kms/common.rs::VaultTestEnvironment`.
## Debugging Tips
- Use the `CLAUDE DEBUG` log lines (left intentionally verbose) to inspect the RustFS server flow during tests.
- If a test fails with `Operation not permitted`, rerun with sandbox overrides (`cargo test ...` with elevated permissions) as shown above.
- Attach `RUST_LOG=rustfs::kms=debug` to surface detailed backend interactions.
## CI Recommendations
- Split KMS tests into a dedicated job so slower suites (Vault) do not gate unrelated changes.
- Cache the Vault binary and reuse it across runs to minimise setup time.
- Surface logs and `target/debug/e2e_test-*` binaries as artifacts when failures occur.
For API usage examples and configuration reference, consult [http-api.md](http-api.md) and [dynamic-configuration-guide.md](dynamic-configuration-guide.md).
docs/kms/troubleshooting.md
+55
View File
@@ -0,0 +1,55 @@
# KMS Troubleshooting
Use this checklist to diagnose and resolve common KMS-related issues.
## Quick Diagnostics
1. **Check status**
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
http://localhost:9000/rustfs/admin/v3/kms/status
```
2. **Inspect logs** – enable `RUST_LOG=rustfs::kms=debug`.
3. **Verify backend reachability** – for Vault, run `vault status` and test network connectivity.
4. **Run a smoke test** – upload a small object with `--server-side-encryption AES256`.
## Common Issues
| Symptom | Likely Cause | Resolution |
|---------|--------------|------------|
| `status: NotConfigured` | KMS was never configured or configuration failed. | POST `/kms/configure`, then `/kms/start`. Check logs for JSON parsing errors. |
| `healthy: false` | Backend health probe failed; Vault sealed or filesystem inaccessible. | Unseal Vault, confirm permissions on the key directory, re-run `/kms/start`. |
| `InternalError: failed to create key` | Backend rejected the request (e.g. Vault policy). | Review Vault audit logs and ensure the RustFS policy has `transit/keys/*` access. |
| `AccessDenied` when downloading SSE-C objects | Missing/incorrect SSE-C headers. | Provide the same `x-amz-server-side-encryption-customer-*` headers used during upload. |
| Multipart SSE-C download truncated | Parts smaller than 5 MiB stored inline; older builds mishandled them. | Re-upload with ≥5 MiB parts or upgrade to the latest RustFS build. |
| `Operation not permitted` during tests | OS sandbox blocked launching `vault` or `rustfs`. | Re-run tests with elevated permissions (`cargo test ...` with sandbox overrides). |
| `KMS key directory is required for local backend` | Started RustFS with `--kms-backend local` but no `--kms-key-dir`. | Supply the flag or use the dynamic API to set the directory before calling `/kms/start`. |
## Clearing the Cache
If data keys become stale (e.g. after manual rotation in Vault), clear the cache:
```bash
awscurl --service s3 --region us-east-1 \
--access_key admin --secret_key admin \
-X POST http://localhost:9000/rustfs/admin/v3/kms/clear-cache
```
## Resetting the Service
1. `POST /kms/stop`
2. `POST /kms/configure` with the known-good payload
3. `POST /kms/start`
4. Verify with `/kms/status`
## Support Data Collection
When opening an issue, capture:
- Output of `/kms/status` and `/kms/config`
- Relevant RustFS logs (`rustfs::kms=*`)
- Vault audit log snippets (if using Vault)
- The SSE headers used by the failing client request
Providing these artifacts drastically speeds up triage.