Compare commits

..

1 Commits

Author SHA1 Message Date
overtrue c568a54797 fix(replication): honor disabled version deletes 2026-07-30 13:37:01 +08:00
203 changed files with 4349 additions and 19472 deletions
+16 -80
View File
@@ -1,21 +1,19 @@
---
name: rustfs-release-publish
description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
bump version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify preview Release assets -> run binary locally + console checks
-> verify release artifacts -> run binary locally + console checks
-> validate with latest rc client
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
@@ -25,7 +23,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
@@ -47,18 +45,14 @@ Rules:
## Preview tag naming
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N``build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -69,61 +63,6 @@ Rules:
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
```bash
CONSOLE_REPO="rustfs/console"
CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
--jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
```
- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
```bash
CONSOLE_SCRATCH=$(mktemp -d)
gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
```
If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
```bash
git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
```
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
3. Find the exact tag run and wait for completion:
```bash
gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
```
4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
```bash
gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
```
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -146,17 +85,16 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and preview Release verification
## Phase 3 — CI and artifact verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
- `isPrerelease` must be `true`.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
## Phase 4 — Run the artifact locally, verify the console
@@ -219,15 +157,13 @@ git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
-7
View File
@@ -23,8 +23,6 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
pull_request:
types: [ opened, synchronize, reopened, closed ]
@@ -35,8 +33,6 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
schedule:
- cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
@@ -100,9 +96,6 @@ jobs:
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+81 -50
View File
@@ -107,21 +107,13 @@ jobs:
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
# Tag push - release or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Preview tags publish a GitHub prerelease for validation, but
# must not update any latest channel.
if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Preview build detected: $tag_name"
elif [[ "$tag_name" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
@@ -722,10 +714,6 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
@@ -744,9 +732,7 @@ jobs:
echo ""
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -754,11 +740,11 @@ jobs:
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release for every valid release tag, including previews
# Create GitHub Release (only for tag pushes)
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
permissions:
contents: write
@@ -781,12 +767,9 @@ jobs:
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
@@ -800,24 +783,54 @@ jobs:
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
else
TITLE="RustFS $VERSION"
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
echo "Created release: $RELEASE_URL"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
permissions:
contents: write
@@ -907,8 +920,8 @@ jobs:
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
steps:
- name: Update latest.json
@@ -967,33 +980,51 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Publish release
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Update release notes and publish
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
RELEASE_TYPE="release"
fi
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
+1 -9
View File
@@ -82,8 +82,7 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
outputs:
should_build: ${{ steps.check.outputs.should_build }}
@@ -221,13 +220,6 @@ jobs:
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
+2 -3
View File
@@ -33,12 +33,11 @@ jobs:
build-helm-package:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
github.event_name == 'workflow_dispatch' ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
contains(github.event.workflow_run.head_branch, '.')
)
outputs:
Generated
+35 -133
View File
@@ -1528,7 +1528,7 @@ version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -1547,7 +1547,7 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -1598,7 +1598,7 @@ version = "3.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dee98b0db6a962de883bf5d20362dee4d7ca0d12fe39a7c6c73c844e1cd7c1f"
dependencies = [
"darling 0.20.11",
"darling 0.23.0",
"ident_case",
"prettyplease",
"proc-macro2",
@@ -1870,7 +1870,7 @@ version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common 0.1.7",
"crypto-common 0.1.6",
"inout 0.1.4",
]
@@ -2328,7 +2328,7 @@ version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"rand_core 0.6.4",
"subtle",
"zeroize",
@@ -2353,11 +2353,11 @@ dependencies = [
[[package]]
name = "crypto-common"
version = "0.1.7"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"typenum",
]
@@ -3554,7 +3554,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer 0.10.4",
"const-oid 0.9.6",
"crypto-common 0.1.7",
"crypto-common 0.1.6",
"subtle",
]
@@ -3672,7 +3672,6 @@ dependencies = [
"flate2",
"futures",
"hex",
"hotpath",
"http 1.5.0",
"http-body-util",
"hyper",
@@ -3814,7 +3813,7 @@ dependencies = [
"crypto-bigint 0.5.5",
"digest 0.10.7",
"ff 0.13.1",
"generic-array 0.14.7",
"generic-array 0.14.9",
"group 0.13.0",
"hkdf 0.12.4",
"pem-rfc7468 0.7.0",
@@ -4259,9 +4258,9 @@ dependencies = [
[[package]]
name = "generic-array"
version = "0.14.7"
version = "0.14.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2"
dependencies = [
"typenum",
"version_check",
@@ -4274,7 +4273,7 @@ version = "1.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab4e5aa225bc56696909483320f0ff9b600f1a971b52e07a17d70f3d9b43254b"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"rustversion",
"typenum",
]
@@ -4373,9 +4372,9 @@ dependencies = [
[[package]]
name = "google-cloud-auth"
version = "1.15.0"
version = "1.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f54aab44c16b8463ae11b165a87c3d484780231f157bb1ed65843d591beb5abd"
checksum = "a3494870d06f3cbbb3561ada6f234982549e3a2fb31e719ef258e6eadb9ae09a"
dependencies = [
"async-trait",
"aws-lc-rs",
@@ -4402,9 +4401,9 @@ dependencies = [
[[package]]
name = "google-cloud-gax"
version = "1.13.0"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9a46dd0fd026bbc4a5d84e6ab0c941cee6e3b057976a0bb107fdb5238ce598f"
checksum = "3103a4a9013f1aed573ca56e19a9680b0211643a99ea85caf524b397d6be8be3"
dependencies = [
"bytes",
"futures",
@@ -4421,9 +4420,9 @@ dependencies = [
[[package]]
name = "google-cloud-gax-internal"
version = "0.7.16"
version = "0.7.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb04c54317ace06d489213f761797240b3046142a9b7ce6b9a82a9d134e193d1"
checksum = "c0df265fba091ed7e00ecd0755009423310163f8b52820f007b6b4d97f4c6617"
dependencies = [
"bytes",
"futures",
@@ -4525,9 +4524,9 @@ dependencies = [
[[package]]
name = "google-cloud-storage"
version = "1.17.0"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9227f65175fa91a6e41f246797917697efdadfe09dd8ea84ad8b737a71efbd28"
checksum = "dc4b1d78c88db5c2530b12461e373a7d0d3a6caa3f6c1fc14e5d824cf2aeb307"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -4578,9 +4577,9 @@ dependencies = [
[[package]]
name = "google-cloud-wkt"
version = "1.7.0"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7fccf98cfd5481a5f5a285181ab0c62123d7d47cd2bb7299448440649349e4e7"
checksum = "46df1fcc3ab69164af3f4199ed21f45b5dbc56d9f03211eb4fa20116d442364b"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -4917,31 +4916,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66750a77f4f6b408a148be5102ef1f3ba7172def7ee92b1cfc75d9f7a3870453"
dependencies = [
"arc-swap",
"async-channel",
"async-trait",
"cfg-if",
"crossbeam-channel",
"flate2",
"futures-channel",
"futures-util",
"hdrhistogram",
"hotpath-macros",
"hotpath-meta",
"http 1.5.0",
"libc",
"object 0.36.7",
"parking_lot",
"pin-project-lite",
"prettytable-rs",
"quanta",
"regex",
"reqwest",
"reqwest-middleware",
"rustc-demangle",
"serde",
"serde_json",
"tiny_http",
"tokio",
]
[[package]]
@@ -4955,21 +4942,6 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "hotpath-macros-meta"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "21f2f70b29b6f42311acd2fb0b2f91a34d9a573c76fb8a7f51970670a1673a49"
[[package]]
name = "hotpath-meta"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b771f77b2f409086bb40ff029f850ce99d17118d4e207df0f28cb32bd7568c7"
dependencies = [
"hotpath-macros-meta",
]
[[package]]
name = "htmlescape"
version = "0.3.1"
@@ -5051,9 +5023,9 @@ checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15"
[[package]]
name = "hybrid-array"
version = "0.4.14"
version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b"
checksum = "818356c5132c1fede50f837ca96afbe78ff42413047f4abb886217845e1b6c8c"
dependencies = [
"ctutils",
"subtle",
@@ -5301,7 +5273,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"block-padding 0.3.3",
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -6811,15 +6783,6 @@ dependencies = [
"objc2-core-foundation",
]
[[package]]
name = "object"
version = "0.36.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87"
dependencies = [
"memchr",
]
[[package]]
name = "object"
version = "0.37.3"
@@ -7857,7 +7820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck",
"itertools 0.13.0",
"itertools 0.10.5",
"log",
"multimap",
"once_cell",
@@ -7877,7 +7840,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck",
"itertools 0.13.0",
"itertools 0.10.5",
"log",
"multimap",
"petgraph 0.8.3",
@@ -7898,7 +7861,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.13.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -7911,7 +7874,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.13.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8551,21 +8514,6 @@ dependencies = [
"web-sys",
]
[[package]]
name = "reqwest-middleware"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07bc3f1384cffa4f274dad2d4ddd73aed32fed8f786d96c6be8aa4e5fd3c3b58"
dependencies = [
"anyhow",
"async-trait",
"http 1.5.0",
"reqwest",
"serde",
"thiserror 2.0.19",
"tower-service",
]
[[package]]
name = "resolv-conf"
version = "0.7.6"
@@ -9040,7 +8988,6 @@ dependencies = [
"const-str",
"futures",
"hashbrown 0.17.1",
"hotpath",
"metrics",
"rustfs-config",
"rustfs-s3-types",
@@ -9061,7 +9008,6 @@ dependencies = [
"base64-simd",
"bytes",
"crc-fast",
"hotpath",
"http 1.5.0",
"md-5 0.11.0",
"pretty_assertions",
@@ -9075,7 +9021,6 @@ name = "rustfs-common"
version = "1.0.0-beta.12"
dependencies = [
"chrono",
"hotpath",
"metrics",
"rmp-serde",
"s3s",
@@ -9090,7 +9035,6 @@ dependencies = [
name = "rustfs-concurrency"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"insta",
"rustfs-io-core",
"serde",
@@ -9104,7 +9048,6 @@ name = "rustfs-config"
version = "1.0.0-beta.12"
dependencies = [
"const-str",
"hotpath",
"serde",
"serde_json",
]
@@ -9115,7 +9058,6 @@ version = "1.0.0-beta.12"
dependencies = [
"base64-simd",
"hmac 0.13.0",
"hotpath",
"rand 0.10.2",
"serde",
"serde_json",
@@ -9131,7 +9073,6 @@ dependencies = [
"argon2",
"base64-simd",
"chacha20poly1305",
"hotpath",
"jsonwebtoken 11.0.0",
"pbkdf2 0.13.0",
"rand 0.10.2",
@@ -9149,7 +9090,6 @@ name = "rustfs-data-usage"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"rmp-serde",
"rustfs-filemeta",
"serde",
@@ -9159,6 +9099,7 @@ dependencies = [
name = "rustfs-ecstore"
version = "1.0.0-beta.12"
dependencies = [
"aes-gcm",
"arc-swap",
"async-channel",
"async-recursion",
@@ -9174,6 +9115,7 @@ dependencies = [
"byteorder",
"bytes",
"bytesize",
"chacha20poly1305",
"chrono",
"criterion",
"enumset",
@@ -9227,6 +9169,7 @@ dependencies = [
"rustfs-erasure-codec",
"rustfs-filemeta",
"rustfs-io-metrics",
"rustfs-kms",
"rustfs-lifecycle",
"rustfs-lock",
"rustfs-madmin",
@@ -9295,7 +9238,6 @@ dependencies = [
name = "rustfs-extension-schema"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"serde",
"serde_json",
"thiserror 2.0.19",
@@ -9334,7 +9276,6 @@ dependencies = [
"async-trait",
"base64 0.23.0",
"futures",
"hotpath",
"http 1.5.0",
"metrics",
"rustfs-common",
@@ -9366,7 +9307,6 @@ dependencies = [
"async-trait",
"base64-simd",
"futures",
"hotpath",
"http 1.5.0",
"jsonwebtoken 11.0.0",
"moka",
@@ -9401,7 +9341,6 @@ name = "rustfs-io-core"
version = "1.0.0-beta.12"
dependencies = [
"bytes",
"hotpath",
"memmap2",
"rustfs-io-metrics",
"thiserror 2.0.19",
@@ -9414,13 +9353,10 @@ name = "rustfs-io-metrics"
version = "1.0.0-beta.12"
dependencies = [
"criterion",
"hotpath",
"metrics",
"metrics-util",
"num_cpus",
"rustfs-common",
"rustfs-s3-ops",
"rustfs-utils",
"sysinfo",
"thiserror 2.0.19",
"tokio",
@@ -9481,7 +9417,6 @@ version = "1.0.0-beta.12"
dependencies = [
"bytes",
"futures",
"hotpath",
"http 1.5.0",
"http-body 1.1.0",
"http-body-util",
@@ -9507,25 +9442,20 @@ name = "rustfs-kms"
version = "1.0.0-beta.12"
dependencies = [
"aes-gcm",
"anyhow",
"arc-swap",
"argon2",
"async-trait",
"base64 0.23.0",
"chacha20poly1305",
"hex",
"hotpath",
"insta",
"jiff",
"md-5 0.11.0",
"metrics",
"metrics-util",
"moka",
"rand 0.10.2",
"reqwest",
"rustfs-security-governance",
"rustfs-utils",
"rustify",
"serde",
"serde_json",
"sha2 0.11.0",
@@ -9534,7 +9464,6 @@ dependencies = [
"tempfile",
"thiserror 2.0.19",
"tokio",
"tokio-util",
"tracing",
"url",
"uuid",
@@ -9547,7 +9476,6 @@ name = "rustfs-lifecycle"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"metrics",
"metrics-util",
"proptest",
@@ -9572,7 +9500,6 @@ dependencies = [
"async-trait",
"crossbeam-queue",
"futures",
"hotpath",
"parking_lot",
"rand 0.10.2",
"rustfs-io-metrics",
@@ -9594,7 +9521,6 @@ version = "1.0.0-beta.12"
dependencies = [
"chrono",
"flate2",
"hotpath",
"regex",
"serde",
"serde_json",
@@ -9612,7 +9538,6 @@ name = "rustfs-madmin"
version = "1.0.0-beta.12"
dependencies = [
"chrono",
"hotpath",
"humantime",
"hyper",
"rmp-serde",
@@ -9633,7 +9558,6 @@ dependencies = [
"criterion",
"form_urlencoded",
"hashbrown 0.17.1",
"hotpath",
"metrics",
"percent-encoding",
"quick-xml",
@@ -9663,7 +9587,6 @@ version = "1.0.0-beta.12"
dependencies = [
"criterion",
"futures",
"hotpath",
"rustfs-config",
"rustfs-io-metrics",
"rustfs-utils",
@@ -9682,7 +9605,6 @@ version = "1.0.0-beta.12"
dependencies = [
"bytes",
"criterion",
"hotpath",
"metrics",
"metrics-util",
"moka",
@@ -9705,7 +9627,6 @@ dependencies = [
"flate2",
"futures-util",
"glob",
"hotpath",
"jiff",
"libc",
"metrics",
@@ -9754,7 +9675,6 @@ dependencies = [
"base64-simd",
"chrono",
"futures",
"hotpath",
"ipnetwork",
"jsonwebtoken 11.0.0",
"moka",
@@ -9791,7 +9711,6 @@ dependencies = [
"futures-util",
"hex",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"http-body-util",
"hyper",
@@ -9844,7 +9763,6 @@ name = "rustfs-protos"
version = "1.0.0-beta.12"
dependencies = [
"flatbuffers",
"hotpath",
"prost 0.14.4",
"rmp-serde",
"rustfs-common",
@@ -9869,7 +9787,6 @@ version = "1.0.0-beta.12"
dependencies = [
"byteorder",
"bytes",
"hotpath",
"regex",
"rmp",
"rmp-serde",
@@ -9928,7 +9845,6 @@ dependencies = [
"chacha20poly1305",
"hex",
"hmac 0.13.0",
"hotpath",
"minlz",
"pin-project-lite",
"rand 0.10.2",
@@ -9946,7 +9862,6 @@ dependencies = [
name = "rustfs-s3-ops"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"rustfs-s3-types",
]
@@ -9954,7 +9869,6 @@ dependencies = [
name = "rustfs-s3-types"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"serde",
"serde_json",
]
@@ -9969,7 +9883,6 @@ dependencies = [
"datafusion",
"futures",
"futures-core",
"hotpath",
"http 1.5.0",
"metrics",
"parking_lot",
@@ -9998,7 +9911,6 @@ dependencies = [
"datafusion",
"derive_builder",
"futures",
"hotpath",
"parking_lot",
"rustfs-s3select-api",
"s3s",
@@ -10016,7 +9928,6 @@ dependencies = [
"futures",
"hex-simd",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"metrics",
"rand 0.10.2",
@@ -10049,7 +9960,6 @@ dependencies = [
name = "rustfs-security-governance"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"thiserror 2.0.19",
]
@@ -10059,7 +9969,6 @@ version = "1.0.0-beta.12"
dependencies = [
"base64-simd",
"bytes",
"hotpath",
"http 1.5.0",
"hyper",
"rustfs-utils",
@@ -10076,7 +9985,6 @@ name = "rustfs-storage-api"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"insta",
"rustfs-filemeta",
"serde",
@@ -10098,7 +10006,6 @@ dependencies = [
"deadpool-postgres",
"futures-util",
"hashbrown 0.17.1",
"hotpath",
"hyper",
"hyper-rustls",
"lapin",
@@ -10144,7 +10051,6 @@ dependencies = [
name = "rustfs-test-utils"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-storage-api",
@@ -10161,7 +10067,6 @@ name = "rustfs-tls-runtime"
version = "1.0.0-beta.12"
dependencies = [
"arc-swap",
"hotpath",
"metrics",
"rcgen",
"rustfs-common",
@@ -10183,7 +10088,6 @@ version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"axum",
"hotpath",
"http 1.5.0",
"ipnetwork",
"metrics",
@@ -10230,7 +10134,6 @@ dependencies = [
"hex-simd",
"highway",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"hyper",
"local-ip-address",
@@ -10263,7 +10166,6 @@ dependencies = [
"astral-tokio-tar",
"async-compression",
"criterion",
"hotpath",
"tempfile",
"thiserror 2.0.19",
"tokio",
@@ -10607,7 +10509,7 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
dependencies = [
"base16ct 0.2.0",
"der 0.7.10",
"generic-array 0.14.7",
"generic-array 0.14.9",
"pkcs8 0.10.2",
"subtle",
"zeroize",
@@ -11597,7 +11499,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
+3 -4
View File
@@ -250,8 +250,8 @@ enumset = "1.1.14"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.4"
google-cloud-storage = "1.17.0"
google-cloud-auth = "1.15.0"
google-cloud-storage = "1.16.0"
google-cloud-auth = "1.14.0"
hashbrown = { version = "0.17.1" }
hex = "0.4.3"
hex-simd = "0.8.0"
@@ -285,7 +285,6 @@ reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.5.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
@@ -348,7 +347,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1.52"
hotpath = { version = "0.22.0", default-features = false }
hotpath = "0.22.0"
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
-26
View File
@@ -25,33 +25,7 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
keywords = ["audit", "target", "management", "fan-out", "RustFS"]
categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-config/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-targets/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-targets/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-targets/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-targets = { workspace = true }
rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
rustfs-s3-types = { workspace = true }
-7
View File
@@ -28,14 +28,7 @@ documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
-7
View File
@@ -27,14 +27,7 @@ categories = ["web-programming", "development-tools", "data-structures"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
-7
View File
@@ -13,14 +13,7 @@ categories = ["concurrency", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
# Internal crates
rustfs-io-core = { workspace = true }
serde = { workspace = true, features = ["derive"] }
-4
View File
@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,6 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
-7
View File
@@ -24,14 +24,7 @@ description = "Credentials management utilities for RustFS, enabling secure hand
keywords = ["rustfs", "Minio", "credentials", "authentication", "authorization"]
categories = ["web-programming", "development-tools", "data-structures", "security"]
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
-4
View File
@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -50,9 +49,6 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
fips = []
crypto = [
"dep:aes-gcm",
-7
View File
@@ -27,14 +27,7 @@ categories = ["data-structures", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "rustfs-filemeta/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-filemeta/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
async-trait = { workspace = true }
-48
View File
@@ -25,58 +25,10 @@ workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-protos/hotpath",
"rustfs-rio/hotpath",
"rustfs-signer/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
ftps = []
sftp = []
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-credentials.workspace = true
rustfs-ecstore.workspace = true
@@ -2136,20 +2136,11 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
assert_eq!(terminal["bucket"].as_str(), Some(bucket.as_str()));
assert_eq!(terminal["prefix"].as_str(), Some(prefix));
assert_eq!(terminal["dry_run"].as_bool(), Some(false));
let terminal_status = terminal["status"].as_str();
assert!(
matches!(terminal_status, Some("partial" | "unknown")),
assert_eq!(
terminal["status"].as_str(),
Some("partial"),
"small transition queue should surface terminal backpressure: {terminal}"
);
if terminal_status == Some("unknown") {
let failure_reason = terminal["failure_reason"]
.as_str()
.ok_or_else(|| format!("unknown terminal status omitted failure_reason: {terminal}"))?;
assert!(
failure_reason.contains("worker result was not persisted before the transition queue drained"),
"unknown terminal status should identify lost worker-result persistence: {terminal}"
);
}
let skipped_queue_full = terminal["report"]["skipped_queue_full"]
.as_u64()
.ok_or_else(|| format!("terminal status omitted report.skipped_queue_full: {terminal}"))?;
@@ -3237,6 +3237,116 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
.await?;
assert_eq!(retained.body.collect().await?.into_bytes().as_ref(), b"versioned replication payload v2");
source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.version_id(delete_marker_version_id)
.send()
.await?;
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let target_state = list_replication_state(&target_client, target_bucket).await?;
assert!(
target_state.iter().all(|entry| entry.version_id != delete_marker_version_id),
"target retained the explicitly purged delete-marker version"
);
assert!(
target_state
.iter()
.any(|entry| !entry.delete_marker && entry.version_id == retained_version_id),
"target removed the retained object version while purging the delete marker: {target_state:?}"
);
Ok(())
}
#[tokio::test]
#[serial]
async fn test_bucket_replication_disabled_version_delete_preserves_target_versions_issue_5442() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_env_vars = replication_fast_env();
source_env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_env.start_rustfs_server_with_env(vec![], &source_env_vars).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env.start_rustfs_server_without_cleanup(vec![]).await?;
let source_bucket = "replication-no-version-delete-src";
let target_bucket = "replication-no-version-delete-dst";
let object_key = "retained-versions.txt";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication_with_delete_statuses(&source_env, source_bucket, &target_arn, "Enabled", None).await?;
let put = source_client
.put_object()
.bucket(source_bucket)
.key(object_key)
.body(ByteStream::from_static(b"permanent delete replication disabled"))
.send()
.await?;
let object_version_id = put.version_id().ok_or("source PUT omitted version ID")?.to_string();
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let delete = source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.send()
.await?;
let delete_marker_version_id = delete
.version_id()
.ok_or("source DELETE omitted marker version ID")?
.to_string();
assert_eq!(delete.delete_marker(), Some(true));
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let expected_target_state = list_replication_state(&target_client, target_bucket).await?;
assert_eq!(expected_target_state.len(), 2);
assert!(
expected_target_state
.iter()
.any(|entry| !entry.delete_marker && entry.version_id == object_version_id)
);
assert!(
expected_target_state
.iter()
.any(|entry| entry.delete_marker && entry.version_id == delete_marker_version_id)
);
for version_id in [&object_version_id, &delete_marker_version_id] {
source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.version_id(version_id)
.send()
.await?;
}
assert!(list_replication_state(&source_client, source_bucket).await?.is_empty());
let observation_deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
let target_state = list_replication_state(&target_client, target_bucket).await?;
assert_eq!(
target_state, expected_target_state,
"disabled permanent-delete replication changed target versions"
);
if tokio::time::Instant::now() >= observation_deadline {
break;
}
sleep(Duration::from_millis(100)).await;
}
Ok(())
}
+5 -86
View File
@@ -33,98 +33,14 @@ workspace = true
[features]
default = []
rio-v2 = ["dep:rustfs-rio-v2"]
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/async-channel",
"hotpath/parking_lot",
"hotpath/reqwest-0-13",
"rustfs-checksums/hotpath",
"rustfs-common/hotpath",
"rustfs-concurrency/hotpath",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-io-metrics/hotpath",
"rustfs-lifecycle/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-object-capacity/hotpath",
"rustfs-policy/hotpath",
"rustfs-protos/hotpath",
"rustfs-replication/hotpath",
"rustfs-rio/hotpath",
"rustfs-rio-v2?/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-signer/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-tls-runtime/hotpath",
"rustfs-utils/hotpath",
"rustfs-crypto/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-checksums/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-concurrency/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
"rustfs-lifecycle/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-object-capacity/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-replication/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-rio-v2?/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-tls-runtime/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-crypto/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-checksums/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-concurrency/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
"rustfs-lifecycle/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-object-capacity/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-replication/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-rio-v2?/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-tls-runtime/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-crypto/hotpath-cpu",
]
hotpath = ["dep:hotpath", "hotpath/hotpath", "rustfs-filemeta/hotpath", "rustfs-rio/hotpath"]
# Exposes shared lifecycle/tier test utilities (MockWarmBackend, fault
# injection, xl.meta transition assertions) via `api::tier::test_util`.
# Enable only from `[dev-dependencies]` (rustfs/backlog#1148 ilm-6).
test-util = []
[dependencies]
hotpath.workspace = true
hotpath = { workspace = true, optional = true }
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
rustfs-rio.workspace = true
@@ -141,6 +57,7 @@ rustfs-policy.workspace = true
rustfs-protos.workspace = true
rustfs-replication.workspace = true
rustfs-lifecycle.workspace = true
rustfs-kms.workspace = true
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-object-capacity.workspace = true
@@ -208,6 +125,8 @@ libc.workspace = true
rustix = { workspace = true, features = ["process", "fs"] }
rustfs-madmin.workspace = true
reqwest = { workspace = true }
aes-gcm = { workspace = true, features = ["rand_core"] }
chacha20poly1305.workspace = true
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
smallvec = { workspace = true, features = ["serde"] }
+4 -6
View File
@@ -391,12 +391,10 @@ pub mod notification {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
unregister_object_mutation_hook,
BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
};
pub use crate::store::PreparedGetObjectReader;
}
@@ -1278,6 +1278,12 @@ fn build_remove_object_headers(version_id: Option<&str>, opts: &RemoveObjectOpti
fn resolve_delete_api_version_id(version_id: Option<String>, opts: &RemoveObjectOptions) -> Option<String> {
if opts.replication_request && opts.replication_delete_marker {
None
} else if opts.replication_request
&& version_id
.as_deref()
.is_some_and(|version_id| Uuid::parse_str(version_id).is_ok_and(|version_id| version_id.is_nil()))
{
Some("null".to_string())
} else {
version_id
}
@@ -2259,6 +2265,12 @@ mod tests {
assert_eq!(got.as_deref(), Some(vid.as_str()));
}
#[test]
fn null_version_purge_sends_s3_null_version_id() {
let got = resolve_delete_api_version_id(Some(Uuid::nil().to_string()), &remove_opts(true, false));
assert_eq!(got.as_deref(), Some("null"));
}
#[test]
fn delete_marker_propagation_omits_versionid_query_param() {
// Propagating a delete-marker CREATION (delete_marker=true): the target
+6 -93
View File
@@ -579,26 +579,8 @@ pub struct BucketMetadataSys {
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
/// Deduplicates concurrent lazy loads of one bucket's metadata, so N
/// simultaneous cache misses issue a single disk read instead of N.
///
/// This is the `singleflight` that upstream applies to its own lazy
/// `GetConfig`. Without it the namespace *read* lock the load holds is no
/// help: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers, and every caller still pays a full
/// erasure-set metadata fanout. A separate registry from
/// `metadata_publish_locks`, reusing the same per-bucket lock machinery.
///
/// Lock order: this lock, then the namespace lock, then the publish lock,
/// then the metadata map. It is only ever taken as the first of those, so
/// it cannot invert against a path that already holds one of the others.
lazy_load_locks: Arc<MetadataPublishLockRegistry>,
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool,
/// Counts disk loads taken by the lazy `get_config` path, so a test can
/// prove concurrent misses collapse into one.
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize,
/// Buckets recently observed to have no persisted metadata. Serving the
/// fabricated default from here (instead of re-reading disk) keeps the
/// per-request cost of repeated lookups for such names bounded — without
@@ -617,13 +599,8 @@ impl BucketMetadataSys {
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
lazy_load_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool::new(false),
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize::new(0),
absent_metadata: moka::future::Cache::builder()
.max_capacity(ABSENT_BUCKET_METADATA_MAX_ENTRIES)
.time_to_live(ABSENT_BUCKET_METADATA_TTL)
@@ -638,22 +615,16 @@ impl BucketMetadataSys {
}
fn metadata_publish_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.metadata_publish_locks, bucket)
}
/// Per-bucket gate for the lazy `get_config` disk load. See
/// [`Self::lazy_load_locks`].
fn lazy_load_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.lazy_load_locks, bucket)
}
fn bucket_lock_in(registry: &Arc<MetadataPublishLockRegistry>, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
let mut locks = registry.locks.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let mut locks = self
.metadata_publish_locks
.locks
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
locks.get(bucket).and_then(Weak::upgrade).unwrap_or_else(|| {
let lock = Arc::new_cyclic(|lock| {
Mutex::new(MetadataPublishLockState {
bucket: bucket.to_string(),
registry: Arc::downgrade(registry),
registry: Arc::downgrade(&self.metadata_publish_locks),
lock: lock.clone(),
})
});
@@ -1108,27 +1079,6 @@ impl BucketMetadataSys {
return Ok((Arc::new(bm), true));
}
// Collapse concurrent misses for this bucket into one disk load.
// Taken before the namespace lock — see `lazy_load_locks` for the
// ordering rule.
let load_lock = self.lazy_load_lock(bucket);
let _load_guard = load_lock.lock_owned().await;
// Re-check both caches: whoever held the gate before us may have
// already answered this exact question, and repeating the fanout
// is the whole cost this gate exists to avoid.
if let Some(bm) = self.metadata_map.read().await.get(bucket).cloned() {
return Ok((bm, true));
}
if self.absent_metadata.get(bucket).await.is_some() {
let mut bm = BucketMetadata::new(bucket);
bm.default_timestamps();
return Ok((Arc::new(bm), true));
}
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let lock = self.api.new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -1481,43 +1431,6 @@ mod tests {
use serial_test::serial;
use tokio::time::timeout;
/// Concurrent cache misses for one bucket must collapse into a single disk
/// load.
///
/// The namespace read lock the lazy path already holds does not provide
/// this: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers. Without the dedup gate every caller pays its
/// own namespace-lock acquisition plus a full erasure-set metadata fanout —
/// and the paths that reach `get_config` are per-request, so the multiplier
/// is request concurrency.
#[tokio::test]
async fn concurrent_lazy_loads_of_one_bucket_issue_a_single_disk_read() {
use std::sync::atomic::Ordering;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(BucketMetadataSys::new(ecstore));
// A name with no persisted metadata: every caller misses the map, and
// the absent-cache entry does not exist until the first load records it.
let bucket = "singleflight-bucket";
let waiters = 8;
let results = futures::future::join_all((0..waiters).map(|_| {
let sys = Arc::clone(&sys);
async move { sys.get_config(bucket).await.map(|(bm, _)| bm.name.clone()) }
}))
.await;
for result in results {
assert_eq!(result.expect("every caller must get an answer"), bucket);
}
assert_eq!(
sys.lazy_disk_loads.load(Ordering::Relaxed),
1,
"concurrent misses for one bucket must share a single disk load"
);
}
/// Pins the fail-closed caching contract of the lazy `get_config` path
/// and the refresh no-replace rule: fabricated defaults are returned but
/// never served by the map-only `get()`, persisted metadata is cached on
@@ -13,6 +13,6 @@
// limitations under the License.
pub use rustfs_replication::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, delete_replication_target_arns,
replication_target_arns, should_remove_replication_target, validate_replication_config_target_arns,
};
@@ -28,7 +28,7 @@ use super::replication_logging::{EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED, LOG_COMPONE
use super::replication_metadata_boundary::ReplicationMetadataStore;
use super::replication_object_decision_boundary::{
MustReplicateOptions, ReplicationDeleteSource, ReplicationResyncTargetObject, delete_replication_missing_source_decision,
delete_replication_object_opts, resync_target_for_object,
delete_replication_object_opts, delete_replication_version_id, resync_target_for_object,
};
use super::replication_storage_boundary::{ObjectInfo, ObjectOptions, ObjectToDelete, object_to_delete_for_replication};
use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
@@ -73,7 +73,7 @@ impl ReplicationConfig {
if oi.delete_marker {
let opts = ObjectOpts {
name: oi.name.clone(),
version_id: oi.version_id,
version_id: delete_replication_version_id(oi.delete_marker, oi.version_id, !oi.version_purge_status.is_empty()),
delete_marker: true,
op_type: ReplicationType::Delete,
existing_object: true,
@@ -299,8 +299,14 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
#[cfg(test)]
mod tests {
use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
use s3s::dto::{
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ReplicationRule, ReplicationRuleStatus,
};
use uuid::Uuid;
use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
use super::super::replication_target_boundary::BucketTarget;
use super::*;
fn replication_rule() -> ReplicationRule {
@@ -360,4 +366,56 @@ mod tests {
assert!(options.is_replication_request());
assert_eq!(options.user_tags(), "env=prod");
}
#[tokio::test]
async fn resync_distinguishes_delete_marker_creation_from_version_purge() {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
Some(BucketTargets {
targets: vec![BucketTarget {
arn: arn.to_string(),
endpoint: "target.example".to_string(),
..Default::default()
}],
}),
);
let marker_version_id = Uuid::new_v4();
let marker = ObjectInfo {
bucket: "source".to_string(),
name: "object".to_string(),
delete_marker: true,
version_id: Some(marker_version_id),
replication_status: ReplicationStatusType::Pending,
..Default::default()
};
let creation = config
.resync(marker.clone(), ReplicateDecision::default(), &HashMap::new())
.await;
assert!(creation.targets.contains_key(arn));
let purge = config
.resync(
ObjectInfo {
version_purge_status: VersionPurgeStatusType::Pending,
..marker
},
ReplicateDecision::default(),
&HashMap::new(),
)
.await;
assert!(purge.targets.is_empty());
}
}
@@ -13,14 +13,14 @@
// limitations under the License.
pub use rustfs_replication::{
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, delete_replication_state_from_config,
delete_replication_version_id, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source,
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, delete_replication_parts,
delete_replication_state_from_config, delete_replication_version_id, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
};
pub(crate) use rustfs_replication::{
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject,
delete_replication_missing_source_decision, delete_replication_object_opts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, resync_target_for_object,
should_retry_delete_marker_purge,
should_retry_delete_marker_purge, version_purge_target_missing,
};
@@ -67,6 +67,7 @@ const EVENT_REPLICATION_BACKPRESSURE: &str = "replication_backpressure";
const EVENT_REPLICATION_RESYNC_LOAD_SKIPPED: &str = "replication_resync_load_skipped";
const EVENT_REPLICATION_RESYNC_RECOVERED: &str = "replication_resync_recovered";
const EVENT_REPLICATION_MRF_QUEUE_UNAVAILABLE: &str = "replication_mrf_queue_unavailable";
const EVENT_REPLICATION_MRF_ENTRY_SKIPPED: &str = "replication_mrf_entry_skipped";
#[derive(Debug, Default)]
pub struct DurableMrfBacklog {
@@ -660,6 +661,21 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
for entry in entries.iter() {
match entry.op {
MrfOpKind::Delete => {
let Some(delete_parts) = entry.delete_parts_for_replay() else {
debug!(
event = EVENT_REPLICATION_MRF_ENTRY_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
reason = "invalid_delete_version_ids",
"Skipped invalid persisted replication delete"
);
continue;
};
let version_purge_id = delete_parts
.version_id
.or_else(|| delete_parts.delete_marker_version_id.filter(|_| !delete_parts.delete_marker));
// Reconstruct a heal delete and re-queue it. We do NOT call
// get_object_info here because the delete-marker or version may
// already be absent from the local store — that is expected.
@@ -674,15 +690,20 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
version_id: version_purge_id,
delete_marker: delete_parts.delete_marker,
replication_status: if entry.replica {
ReplicationStatusType::Replica
} else {
ReplicationStatusType::Empty
},
..Default::default()
};
let dsc = check_replicate_delete(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
version_id: version_purge_id,
..Default::default()
},
&oi,
@@ -708,9 +729,9 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let dv = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
version_id: delete_parts.version_id,
delete_marker_version_id: delete_parts.delete_marker_version_id,
delete_marker: delete_parts.delete_marker,
delete_marker_mtime,
replication_state: Some(rstate),
..Default::default()
@@ -2333,6 +2354,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
let second = MrfReplicateEntry {
@@ -2446,6 +2468,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
@@ -2479,6 +2502,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(dm_vid),
delete_marker: true,
replica: false,
delete_marker_mtime: Some(mtime_nanos),
};
@@ -2512,6 +2536,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
@@ -2540,6 +2565,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
},
MrfReplicateEntry {
@@ -2551,6 +2577,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(del_dm_vid),
delete_marker: true,
replica: false,
delete_marker_mtime: None,
},
];
@@ -2580,6 +2607,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(obj_entry.op, MrfOpKind::Object);
@@ -2594,6 +2622,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(Uuid::new_v4()),
delete_marker: true,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(del_entry.op, MrfOpKind::Delete);
@@ -2609,6 +2638,7 @@ mod tests {
op: MrfOpKind::default(),
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(legacy_entry.op, MrfOpKind::Object, "legacy default must be Object");
@@ -2672,6 +2702,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
}];
let encoded = encode_mrf_file(&entries).expect("durable MRF backlog should encode");
@@ -2702,6 +2733,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
}])
.expect("invalid persisted entry should still encode for boundary testing");
@@ -13,7 +13,7 @@
// limitations under the License.
use super::replication_bandwidth_boundary;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _, delete_replication_target_arns};
use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Result, is_err_object_not_found, is_err_version_not_found};
use super::replication_event_sink::{EventArgs, send_event, send_local_event};
@@ -29,9 +29,10 @@ use super::replication_metadata_boundary::ReplicationMetadataStore;
use super::replication_msgp_boundary::ReplicationMsgpCodec;
use super::replication_object_config::{ReplicationConfig, check_replicate_delete, get_replication_config, must_replicate};
use super::replication_object_decision_boundary::{
MustReplicateOptions, ReplicationMultipartPartInput, heal_uses_delete_replication_path,
MustReplicateOptions, ReplicationMultipartPartInput, delete_replication_parts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, should_retry_delete_marker_purge,
version_purge_target_missing,
};
use super::replication_queue_boundary::DeletedObjectReplicationInfo;
use super::replication_resync_boundary::ResyncStatusType;
@@ -49,7 +50,7 @@ use super::replication_target_boundary::{
PutObjectOptions, PutObjectPartOptions, ReplicationTargetStore, TargetClient, replication_action_for_target_head,
replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options,
replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size,
replication_put_object_options, replication_target_head_is_newer_null_version,
replication_put_object_options, replication_target_head_is_newer_null_version, replication_target_version_id,
};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
@@ -70,9 +71,8 @@ use rustfs_utils::http::{
AMZ_TAGGING_DIRECTIVE, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, has_internal_suffix, insert_str,
};
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, sip_hash};
#[cfg(test)]
use s3s::dto::ReplicationConfiguration;
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::sync::Arc;
use time::OffsetDateTime;
@@ -97,6 +97,8 @@ const EVENT_RESYNC_TASK_FAILED: &str = "replication_resync_task_failed";
const EVENT_RESYNC_TARGET_OPERATION_FAILED: &str = "replication_resync_target_operation_failed";
const EVENT_RESYNC_RUNTIME_CHANNEL_FAILED: &str = "replication_resync_runtime_channel_failed";
const ERR_REPLICATION_METADATA_COPY_UNSUPPORTED: &str = "metadata-only replication is not implemented";
const ERR_VERSION_PURGE_TARGET_STILL_EXISTS: &str = "target version still exists after replication purge";
const ERR_VERSION_PURGE_MISSING_VERSION_ID: &str = "version purge record is missing version ID";
const REPLICATION_TARGET_OFFLINE_ERROR_MARKERS: &[&str] = &[
"dispatch failure",
"timeouterror",
@@ -786,19 +788,46 @@ impl ReplicationResyncer {
}
if roi.delete_marker || !roi.version_purge_status.is_empty() {
let (version_id, dm_version_id) = if roi.version_purge_status.is_empty() {
(None, roi.version_id)
} else {
(roi.version_id, None)
let Some(parts) =
delete_replication_parts(roi.delete_marker, roi.version_id, !roi.version_purge_status.is_empty())
else {
debug!(
event = EVENT_RESYNC_OBJECT_PROCESSED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket_name,
object = %roi.name,
reason = "version_purge_missing_version_id",
"Failed to process resync version purge"
);
let status = TargetReplicationResyncStatus {
bucket: roi.bucket.clone(),
object: roi.name.clone(),
failed_count: 1,
error: Some(ERR_VERSION_PURGE_MISSING_VERSION_ID.to_string()),
..Default::default()
};
if let Err(err) = results_tx.send(status).await {
error!(
event = EVENT_RESYNC_RUNTIME_CHANNEL_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket_name,
reason = "status_channel_send_failed",
error = %err,
"Failed to send resync status"
);
}
continue;
};
let doi = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: roi.name.clone(),
delete_marker_version_id: dm_version_id,
version_id,
delete_marker_version_id: parts.delete_marker_version_id,
version_id: parts.version_id,
replication_state: roi.replication_state.clone(),
delete_marker: roi.delete_marker,
delete_marker: parts.delete_marker,
delete_marker_mtime: roi.mod_time,
..Default::default()
},
@@ -821,21 +850,41 @@ impl ReplicationResyncer {
};
let reset_id = target_client.reset_id.clone();
let is_version_purge = !roi.version_purge_status.is_empty();
let head_result = head_object_with_proxy_stats(
&bucket_name,
target_client.as_ref(),
&target_client.bucket,
&roi.name,
roi.version_id.map(|v| v.to_string()),
replication_target_version_id(roi.version_id, is_version_purge),
)
.await;
let (size, err) = match head_result {
Ok(_) if is_version_purge => {
st.failed_count += 1;
(0, Some(ERR_VERSION_PURGE_TARGET_STILL_EXISTS.to_string()))
}
Ok(_) => {
st.replicated_count += 1;
st.replicated_size += roi.size;
(roi.size, None)
}
Err(err) if is_version_purge => {
let (is_not_found, code) = err
.as_service_error()
.map(|service_err| (service_err.is_not_found(), service_err.code()))
.unwrap_or((false, None));
let raw_status = err.raw_response().map(|response| response.status().as_u16());
let missing = version_purge_target_missing(is_not_found, code, raw_status);
if missing {
st.replicated_count += 1;
(0, None)
} else {
st.failed_count += 1;
(0, resync_target_error_detail(&err))
}
}
Err(err) if roi.delete_marker => {
// Verifying a replicated delete marker: only a
// definitive 404/NoSuchKey or 405/MethodNotAllowed
@@ -852,7 +901,7 @@ impl ReplicationResyncer {
};
if retryable {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
} else {
st.replicated_count += 1;
(0, None)
@@ -871,17 +920,17 @@ impl ReplicationResyncer {
}
Ok(None) => {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
}
Err(e2) => {
st.failed_count += 1;
(0, Some(e2))
(0, resync_target_error_detail(&e2))
}
}
}
Err(err) => {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
}
};
@@ -911,7 +960,7 @@ impl ReplicationResyncer {
"Processed resync object"
);
}
st.error = err.as_ref().and_then(resync_target_error_detail);
st.error = err;
if cancel_token.is_cancelled() {
return;
@@ -1065,22 +1114,27 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
}
let dsc = if heal_uses_delete_replication_path(oi.delete_marker, &oi.version_purge_status) {
check_replicate_delete(
oi.bucket.as_str(),
&ObjectToDelete {
object_name: oi.name.clone(),
version_id: oi.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned: ReplicationVersioningStore::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: ReplicationVersioningStore::prefix_suspended(&oi.bucket, &oi.name).await,
..Default::default()
},
None,
)
.await
match delete_replication_parts(oi.delete_marker, oi.version_id, !oi.version_purge_status.is_empty()) {
Some(parts) => {
check_replicate_delete(
oi.bucket.as_str(),
&ObjectToDelete {
object_name: oi.name.clone(),
version_id: parts.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned: ReplicationVersioningStore::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: ReplicationVersioningStore::prefix_suspended(&oi.bucket, &oi.name).await,
..Default::default()
},
None,
)
.await
}
None => ReplicateDecision::default(),
}
} else {
must_replicate(
oi.bucket.as_str(),
@@ -1150,7 +1204,6 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
} else {
dobj.delete_object.version_id
};
let _rcfg = match get_replication_config(&bucket).await {
Ok(Some(config)) => config,
Ok(None) => {
@@ -1469,8 +1522,8 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
delete_marker_version_id,
)
.await
&& replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await
{
replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await;
break;
}
tokio::time::sleep(TokioDuration::from_secs(1)).await;
@@ -1600,9 +1653,50 @@ async fn source_delete_marker_missing<S: EcstoreObjectOperations>(
}
}
async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo, dsc: &ReplicateDecision) {
fn delete_marker_purge_target_arns(config: &ReplicationConfiguration, dobj: &DeletedObjectReplicationInfo) -> HashSet<String> {
let replica = dobj
.delete_object
.replication_state
.as_ref()
.is_some_and(|state| state.replica_status == ReplicationStatusType::Replica);
delete_replication_target_arns(config, &dobj.delete_object.object_name, replica)
}
#[derive(Debug, PartialEq, Eq)]
enum DeleteMarkerPurgeConfig {
Apply(HashSet<String>),
Stop,
Retry,
}
fn delete_marker_purge_config<E>(
result: std::result::Result<Option<ReplicationConfiguration>, E>,
dobj: &DeletedObjectReplicationInfo,
) -> DeleteMarkerPurgeConfig {
match result {
Ok(Some(config)) => DeleteMarkerPurgeConfig::Apply(delete_marker_purge_target_arns(&config, dobj)),
Ok(None) => DeleteMarkerPurgeConfig::Stop,
Err(_) => DeleteMarkerPurgeConfig::Retry,
}
}
async fn replicate_delete_marker_purge_to_targets(
bucket: &str,
dobj: &DeletedObjectReplicationInfo,
dsc: &ReplicateDecision,
) -> bool {
let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
return;
return true;
};
let marker_creation_purge_targets = if dobj.delete_object.delete_marker {
match delete_marker_purge_config(get_replication_config(bucket).await, dobj) {
DeleteMarkerPurgeConfig::Apply(targets) => Some(targets),
DeleteMarkerPurgeConfig::Stop => return true,
DeleteMarkerPurgeConfig::Retry => return false,
}
} else {
None
};
for tgt_entry in dsc.targets_map.values() {
@@ -1612,6 +1706,12 @@ async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedOb
if !dobj.target_arn.is_empty() && dobj.target_arn != tgt_entry.arn {
continue;
}
if marker_creation_purge_targets
.as_ref()
.is_some_and(|targets| !targets.contains(&tgt_entry.arn))
{
continue;
}
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
continue;
};
@@ -1625,6 +1725,7 @@ async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedOb
)
.await;
}
true
}
async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &DeletedObjectReplicationInfo, storage: Arc<S>) {
@@ -1851,10 +1952,10 @@ async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &Deleted
}
async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_client: Arc<TargetClient>) -> ReplicatedTargetInfo {
let version_id = if let Some(version_id) = &dobj.delete_object.delete_marker_version_id {
version_id.to_owned()
let version_id = if let Some(version_id) = dobj.delete_object.delete_marker_version_id {
Some(version_id)
} else {
dobj.delete_object.version_id.unwrap_or_default()
dobj.delete_object.version_id
};
let mut rinfo = dobj
@@ -1889,11 +1990,7 @@ async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_cli
return rinfo;
}
let version_id = if version_id.is_nil() {
None
} else {
Some(version_id.to_string())
};
let version_id = replication_target_version_id(version_id, is_version_purge);
if dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
match head_object_with_proxy_stats(
@@ -3143,6 +3240,10 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
#[cfg(test)]
mod tests {
use super::*;
use s3s::dto::{
DeleteReplication, DeleteReplicationStatus, Destination, ReplicaModifications, ReplicaModificationsStatus,
ReplicationRule, ReplicationRuleAndOperator, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
};
use std::collections::HashMap;
use time::OffsetDateTime;
use uuid::Uuid;
@@ -3452,6 +3553,110 @@ mod tests {
);
}
#[test]
fn test_delete_marker_purge_targets_follow_delete_and_replica_modification_rules() {
fn rule(arn: &str, delete_status: &'static str) -> ReplicationRule {
ReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(delete_status),
}),
destination: Destination {
bucket: arn.to_string(),
..Default::default()
},
existing_object_replication: None,
filter: None,
id: Some(arn.to_string()),
prefix: Some("logs/".to_string()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
}
}
let enabled_arn = "arn:rustfs:replication:us-east-1:target:enabled";
let disabled_arn = "arn:rustfs:replication:us-east-1:target:disabled";
let delete_marker_version_id = Uuid::new_v4();
let mut config = ReplicationConfiguration {
role: String::new(),
rules: vec![
rule(enabled_arn, DeleteReplicationStatus::ENABLED),
rule(disabled_arn, DeleteReplicationStatus::DISABLED),
],
};
let mut dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "logs/object.txt".to_string(),
delete_marker: true,
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
..Default::default()
};
assert_eq!(delete_marker_purge_target_arns(&config, &dobj), HashSet::from([enabled_arn.to_string()]));
assert_eq!(
delete_marker_purge_config::<()>(Ok(Some(config.clone())), &dobj),
DeleteMarkerPurgeConfig::Apply(HashSet::from([enabled_arn.to_string()]))
);
dobj.delete_object.replication_state = Some(Default::default());
dobj.delete_object
.replication_state
.as_mut()
.expect("test replication state")
.replica_status = ReplicationStatusType::Replica;
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
config.rules[0].source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: ReplicaModificationsStatus::from_static(ReplicaModificationsStatus::ENABLED),
}),
sse_kms_encrypted_objects: None,
});
assert_eq!(delete_marker_purge_target_arns(&config, &dobj), HashSet::from([enabled_arn.to_string()]));
config.rules[0].prefix = None;
config.rules[0].filter = Some(ReplicationRuleFilter {
tag: Some(Tag {
key: Some("env".to_string()),
value: Some("prod".to_string()),
}),
..Default::default()
});
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
config.rules[0].filter = Some(ReplicationRuleFilter {
and: Some(ReplicationRuleAndOperator {
prefix: Some("logs/".to_string()),
tags: Some(vec![Tag {
key: Some("env".to_string()),
value: Some("prod".to_string()),
}]),
}),
..Default::default()
});
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
}
#[test]
fn test_delete_marker_purge_config_errors_are_retryable() {
let delete_marker_version_id = Uuid::new_v4();
let dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "object.txt".to_string(),
delete_marker: true,
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
..Default::default()
};
assert_eq!(delete_marker_purge_config::<()>(Ok(None), &dobj), DeleteMarkerPurgeConfig::Stop);
assert_eq!(delete_marker_purge_config::<()>(Err(()), &dobj), DeleteMarkerPurgeConfig::Retry);
}
#[test]
fn test_is_retryable_delete_replication_head_error_allows_delete_marker_head_responses() {
assert!(
@@ -31,10 +31,13 @@ use rustfs_utils::http::{
};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use uuid::Uuid;
pub(crate) use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
};
#[cfg(test)]
pub(crate) use crate::bucket::target::BucketTarget;
pub(crate) use crate::bucket::target::BucketTargets;
use super::replication_config_store::ReplicationConfigStore;
@@ -305,6 +308,14 @@ pub(crate) fn replication_target_head_is_newer_null_version(object_info: &Object
target_is_newer_than_source_null_version(&replication_source_object(object_info), &replication_target_object(target))
}
pub(crate) fn replication_target_version_id(version_id: Option<Uuid>, version_purge: bool) -> Option<String> {
match version_id {
Some(version_id) if version_id.is_nil() && version_purge => Some("null".to_string()),
Some(version_id) if !version_id.is_nil() => Some(version_id.to_string()),
_ => None,
}
}
pub(crate) fn replication_delete_remove_options(
delete_marker: bool,
replication_mtime: Option<OffsetDateTime>,
@@ -519,6 +530,18 @@ mod tests {
assert!(force.replication_request);
}
#[test]
fn replication_target_version_id_preserves_null_purges() {
assert_eq!(replication_target_version_id(Some(Uuid::nil()), true).as_deref(), Some("null"));
assert_eq!(replication_target_version_id(Some(Uuid::nil()), false), None);
let version_id = Uuid::new_v4();
assert_eq!(
replication_target_version_id(Some(version_id), true).as_deref(),
Some(version_id.to_string().as_str())
);
}
#[test]
fn replication_complete_multipart_options_sets_actual_size() {
let options = replication_complete_multipart_options("1024".to_string());
+15 -1
View File
@@ -46,6 +46,16 @@ lazy_static! {
m.insert("x-amz-replication-status".to_string(), true);
m
};
static ref SSE_HEADERS: HashMap<String, bool> = {
let mut m = HashMap::new();
m.insert("x-amz-server-side-encryption".to_string(), true);
m.insert("x-amz-server-side-encryption-aws-kms-key-id".to_string(), true);
m.insert("x-amz-server-side-encryption-context".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-algorithm".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key-md5".to_string(), true);
m
};
}
pub fn is_standard_query_value(qs_key: &str) -> bool {
@@ -60,12 +70,16 @@ pub fn is_standard_header(header_key: &str) -> bool {
*SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_sse_header(header_key: &str) -> bool {
*SSE_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_amz_header(header_key: &str) -> bool {
let key = header_key.to_lowercase();
key.starts_with("x-amz-meta-")
|| key.starts_with("x-amz-grant-")
|| key == "x-amz-acl"
|| rustfs_utils::http::is_sse_header(header_key)
|| is_sse_header(header_key)
|| key.starts_with("x-amz-checksum-")
}
-4
View File
@@ -1049,10 +1049,6 @@ impl LocalDiskWrapper {
Ok(())
}
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) {
*self.disk_id.write().await = id;
}
/// Get the current disk ID
pub async fn get_current_disk_id(&self) -> Option<Uuid> {
*self.disk_id.read().await
+2 -121
View File
@@ -5078,7 +5078,7 @@ impl LocalDisk {
Ok((buf, mtime))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn read_metadata_with_dmtime(&self, file_path: impl AsRef<Path>) -> Result<(Vec<u8>, Option<OffsetDateTime>)> {
check_path_length(file_path.as_ref().to_string_lossy().as_ref())?;
@@ -5121,7 +5121,7 @@ impl LocalDisk {
Ok((data, modtime))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn read_all_data(&self, volume: &str, volume_dir: impl AsRef<Path>, file_path: impl AsRef<Path>) -> Result<Vec<u8>> {
// TODO: timeout support
let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?;
@@ -6641,49 +6641,6 @@ impl LocalDisk {
let xl_path = object_dir.join(STORAGE_FORMAT_FILE);
restore_delete_rollback_after_error(object_dir, &xl_path, Some(rollback_dir), volume, object, stage, err).await
}
/// Execute every deferred data-dir deletion pending on `volume` right now,
/// even while snapshot leases are still held. Bucket deletion requires it:
/// a streaming reader defers the physical cleanup of an already-deleted
/// version, and a non-force `delete_volume` would otherwise fail closed
/// with `VolumeNotEmpty` on those remnants even though the bucket is
/// logically empty. The still-active readers keep their open descriptors;
/// only path-based reopens observe the removal.
async fn settle_pending_snapshot_deletes(&self, volume: &str) {
let pending: Vec<(SnapshotLeaseKey, DeleteOptions)> = {
let mut registry = self.snapshot_leases.lock().await;
registry
.entries
.iter_mut()
.filter(|(key, entry)| key.volume == volume && !entry.deleting && entry.pending_delete.is_some())
.map(|(key, entry)| {
entry.deleting = true;
(key.clone(), entry.pending_delete.clone().expect("filtered on Some"))
})
.collect()
};
for (key, opts) in pending {
let result = self.delete_unleased(&key.volume, &key.path, &opts).await;
let mut registry = self.snapshot_leases.lock().await;
match result {
Ok(()) => {
registry.entries.remove(&key);
}
Err(err) => {
if let Some(entry) = registry.entries.get_mut(&key) {
entry.deleting = false;
}
warn!(
volume = %key.volume,
path = %key.path,
error = %err,
"failed to settle deferred data-dir deletion before volume removal"
);
}
}
}
}
}
#[async_trait::async_trait]
@@ -9165,14 +9122,6 @@ impl DiskAPI for LocalDisk {
let p = self.get_bucket_path(volume)?;
let _volume_mutation_guard = os::disk_volume_mutation_lock(&self.root, volume).write_owned().await;
// A streaming reader's snapshot lease defers the physical cleanup of
// data dirs whose version delete already committed. Those remnants are
// logically deleted, so run the parked cleanups now instead of letting
// the non-force removal below fail closed on them (the s3-tests SSE-C
// teardown races exactly this way: DeleteObjects, then DeleteBucket
// while an abandoned GET body still pins the lease).
self.settle_pending_snapshot_deletes(volume).await;
// Non-force removes empty directory remnants children-first with
// non-recursive rmdir calls. A file that exists during the scan, or
// appears before its parent is removed, fails closed with
@@ -16032,74 +15981,6 @@ mod test {
assert!(matches!(disk.read_all(volume, &first_part).await, Err(DiskError::FileNotFound)));
}
#[tokio::test]
async fn delete_volume_settles_lease_deferred_cleanup() {
use tempfile::tempdir;
let root_dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let volume = "snapshot-lease-bucket-delete";
let object = "multipart_enc";
let version_id = Uuid::new_v4();
let data_dir = Uuid::new_v4();
let rollback_dir = Uuid::new_v4();
let data_path = path_join_buf(&[object, &data_dir.to_string()]);
let part = path_join_buf(&[&data_path, "part.1"]);
ensure_test_volume(&disk, volume).await;
disk.write_all(volume, &part, Bytes::from_static(b"payload"))
.await
.expect("shard should be written");
let fi = test_file_info(object, version_id, Some(data_dir), None);
disk.write_all(volume, &path_join_buf(&[object, STORAGE_FORMAT_FILE]), test_meta(fi.clone()).into())
.await
.expect("metadata should be written");
// An abandoned streaming GET pins the data dir with a snapshot lease.
let snapshot = disk
.acquire_snapshot_lease(volume, &data_path)
.await
.expect("snapshot lease should be acquired");
disk.delete_version(
volume,
object,
fi.clone(),
false,
DeleteOptions {
old_data_dir: Some(rollback_dir),
..Default::default()
},
)
.await
.expect("version delete should commit metadata");
disk.delete(
volume,
&format!("{object}/{rollback_dir}"),
DeleteOptions {
recursive: true,
immediate: true,
..Default::default()
},
)
.await
.expect("version delete should schedule physical cleanup");
// The bucket is logically empty; a non-force volume delete must settle
// the deferred data-dir cleanup instead of failing with VolumeNotEmpty.
disk.delete_volume(volume, false)
.await
.expect("bucket delete must not observe lease-deferred remnants");
assert!(matches!(
disk.read_all(volume, &part).await,
Err(DiskError::FileNotFound | DiskError::VolumeNotFound)
));
// The late lease release finds nothing pending and stays idempotent.
disk.release_snapshot_lease(volume, &data_path, snapshot)
.await
.expect("releasing the lease after bucket deletion should be a no-op");
}
#[tokio::test]
async fn version_delete_cleanup_intent_survives_local_disk_restart() {
use tempfile::tempdir;
-78
View File
@@ -132,18 +132,6 @@ pub enum Disk {
Remote(Box<RemoteDisk>),
}
impl Disk {
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
match self {
Disk::Local(local_disk) => {
local_disk.set_disk_id_state(id).await;
Ok(())
}
Disk::Remote(remote_disk) => remote_disk.set_disk_id(id).await,
}
}
}
#[async_trait::async_trait]
impl DiskAPI for Disk {
fn to_string(&self) -> String {
@@ -1564,72 +1552,6 @@ mod tests {
let _ = fs::remove_dir_all(&test_dir).await;
}
#[tokio::test]
#[serial_test::serial]
async fn local_disk_id_state_does_not_publish_to_the_process_registry() {
let local_dir = tempfile::tempdir().expect("local disk tempdir should be created");
let mut endpoint =
Endpoint::try_from(local_dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let local_disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
let disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("local wrapper state should accept a disk ID");
let Disk::Local(local_disk) = &disk else {
panic!("test disk should remain local");
};
assert_eq!(local_disk.get_current_disk_id().await, Some(disk_id));
assert!(
!crate::runtime::global::current_ctx()
.local_disk_id_map()
.read()
.await
.contains_key(&disk_id),
"state-only startup publication must not update the process disk-ID registry"
);
disk.set_disk_id_state(None)
.await
.expect("local wrapper state should clear a disk ID");
assert_eq!(local_disk.get_current_disk_id().await, None);
}
#[tokio::test]
async fn remote_disk_id_state_delegates_some_and_none() {
let mut endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let remote_disk = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
Arc::new(crate::cluster::rpc::TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk should initialize");
let disk = Disk::Remote(Box::new(remote_disk));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("remote state should accept a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), Some(disk_id));
disk.set_disk_id_state(None)
.await
.expect("remote state should clear a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), None);
}
#[tokio::test]
async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
let local_dir = tempfile::tempdir().unwrap();
+3 -3
View File
@@ -103,7 +103,7 @@ where
/// or `out` is larger than one shard. On error `out`'s contents are
/// unspecified but never contain bytes that failed the hash check — the copy
/// happens only after verification.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
let want = out.len();
self.begin_read(want)?;
@@ -303,7 +303,7 @@ where
/// Write a (hash+data) block. Returns the number of data bytes written.
/// Returns an error if called after a short write or if data exceeds shard_size.
#[hotpath::measure(label = "BitrotWriter::write")]
#[cfg_attr(feature = "hotpath", hotpath::measure(label = "BitrotWriter::write"))]
pub async fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
if buf.is_empty() {
return Ok(0);
@@ -455,7 +455,7 @@ pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorith
/// stores those as whole-file bitrot with no interleaved hash, so the size guard
/// on the next line would reject a genuinely healthy part. Reading legacy V1
/// whole-file-bitrot objects would need a separate verification path.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
mut r: R,
want_size: usize,
+2 -2
View File
@@ -691,7 +691,7 @@ impl<R> ParallelReader<R>
where
R: crate::erasure::coding::ShardSource,
{
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read(&mut self) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>) {
// On the reconstruction-verifying GET path, read every live shard reader
// in lockstep so all readers advance one block per stripe and stay
@@ -1505,7 +1505,7 @@ where
}
impl Erasure {
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn decode<W, R>(
&self,
writer: &mut W,
+4 -4
View File
@@ -504,7 +504,7 @@ impl Erasure {
Ok((reader, total))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode<R>(
self: Arc<Self>,
reader: R,
@@ -670,7 +670,7 @@ impl Erasure {
Ok((reader, total))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_batched<R>(
self: Arc<Self>,
mut reader: R,
@@ -798,7 +798,7 @@ impl Erasure {
/// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
/// Reads all data, encodes directly, writes shards sequentially.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_inline_small<R>(
self: Arc<Self>,
reader: R,
@@ -813,7 +813,7 @@ impl Erasure {
/// Fast path for single-block non-inline objects: avoids the producer/consumer
/// pipeline in `encode()` while keeping the same writer/quorum/shutdown semantics.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_single_block_non_inline<R>(
self: Arc<Self>,
reader: R,
+5 -5
View File
@@ -640,7 +640,7 @@ impl Erasure {
/// # Returns
/// A vector of encoded shards as `Bytes`.
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -688,7 +688,7 @@ impl Erasure {
/// Encode owned data, avoiding a copy when the caller already has a heap buffer.
/// Falls back to copying into a new buffer if zero-copy conversion fails.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data_owned(&self, data: Vec<u8>) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -752,7 +752,7 @@ impl Erasure {
/// block), the `resize(need_total_size)` below stays within capacity for every
/// `data_len <= block_size` — both shard-size formulas are monotone in
/// `data_len` — so this function never reallocates the buffer.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data_bytes_mut(&self, mut data_buffer: BytesMut, data_len: usize) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -805,7 +805,7 @@ impl Erasure {
///
/// # Returns
/// Ok if reconstruction succeeds, error otherwise.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
@@ -825,7 +825,7 @@ impl Erasure {
}
/// Decode and reconstruct missing data shards, then regenerate parity shards.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn decode_data_and_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
+1 -3
View File
@@ -12,10 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#![recursion_limit = "256"]
/// Scope-based hotpath measurement for `#[async_trait]` methods, where
/// `#[hotpath::measure]` would only time the boxed-future construction.
/// `#[cfg_attr(feature = "hotpath", hotpath::measure)]` would only time the boxed-future construction.
/// The guard records wall time from this statement until the enclosing
/// (desugared) async block completes, including early returns via `?`.
#[cfg(feature = "hotpath")]
@@ -1,80 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use async_trait::async_trait;
use http::{HeaderMap, HeaderValue};
use std::collections::HashMap;
use std::error::Error;
use std::fmt::{Display, Formatter};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadEncryptionMode {
Direct { base_nonce: [u8; 12] },
Object,
}
pub struct ReadEncryptionMaterial {
pub key_bytes: [u8; 32],
pub mode: ReadEncryptionMode,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EncryptionResolutionErrorKind {
InvalidRequest,
InvalidMetadata,
ServiceUnavailable,
DecryptionFailed,
}
#[derive(Debug)]
pub struct EncryptionResolutionError {
kind: EncryptionResolutionErrorKind,
message: String,
}
impl EncryptionResolutionError {
pub fn new(kind: EncryptionResolutionErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
pub fn kind(&self) -> EncryptionResolutionErrorKind {
self.kind
}
}
impl Display for EncryptionResolutionError {
fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.message)
}
}
impl Error for EncryptionResolutionError {}
pub struct ReadEncryptionRequest<'a> {
pub bucket: &'a str,
pub object: &'a str,
pub metadata: &'a HashMap<String, String>,
pub headers: &'a HeaderMap<HeaderValue>,
}
#[async_trait]
pub trait ObjectEncryptionResolver: Send + Sync {
async fn resolve_read_material(
&self,
request: ReadEncryptionRequest<'_>,
) -> Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError>;
}
-5
View File
@@ -84,7 +84,6 @@ pub(crate) fn legacy_encrypted_range_seek_enabled() -> bool {
}
mod body_cache_hook;
mod encryption;
mod hook_slot;
mod object_mutation_hook;
mod readers;
@@ -99,10 +98,6 @@ pub use body_cache_hook::{
pub(crate) use body_cache_hook::{
get_object_body_cache_hook, get_object_body_cache_hook_suppressed, without_get_object_body_cache_hook,
};
pub use encryption::{
EncryptionResolutionError, EncryptionResolutionErrorKind, ObjectEncryptionResolver, ReadEncryptionMaterial,
ReadEncryptionMode, ReadEncryptionRequest,
};
pub(crate) use object_mutation_hook::notify_object_mutation;
pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
pub use readers::*;
File diff suppressed because it is too large Load Diff
+46 -17
View File
@@ -273,9 +273,29 @@ impl ObjectInfo {
}
pub fn is_encrypted(&self) -> bool {
self.user_defined
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
// Corresponding to the logic in rustfs/src/sse.rs/encryption_material_to_metadata function
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
self.user_defined.keys().any(|key| {
let lower = key.to_ascii_lowercase();
lower.starts_with("x-minio-encryption-")
|| lower.starts_with("x-minio-internal-server-side-encryption-")
|| matches!(
lower.as_str(),
"x-minio-internal-encrypted-multipart"
| "x-rustfs-encryption-key"
| "x-rustfs-encryption-algorithm"
| "x-rustfs-encryption-iv"
| "x-rustfs-encryption-key-id"
| "x-rustfs-encryption-context"
| "x-rustfs-encryption-tag"
| "x-amz-server-side-encryption-aws-kms-key-id"
| SSEC_ALGORITHM_HEADER
| SSEC_KEY_HEADER
| SSEC_KEY_MD5_HEADER
| "x-amz-server-side-encryption"
)
})
}
/// Maximum inline size for non-versioned objects (128 KiB).
@@ -319,7 +339,26 @@ impl ObjectInfo {
}
pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
rustfs_utils::http::get_object_encryption_original_size(&self.user_defined)
let actual_size = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE);
if let Some(size_str) = self
.user_defined
.get("x-rustfs-encryption-original-size")
.map(String::as_str)
.or_else(|| {
self.user_defined
.get("x-amz-server-side-encryption-customer-original-size")
.map(String::as_str)
})
.or(actual_size.as_deref())
&& !size_str.is_empty()
{
let size = size_str
.parse::<i64>()
.map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
return Ok(Some(size));
}
Ok(None)
}
pub fn decrypted_size(&self) -> std::io::Result<i64> {
@@ -349,6 +388,9 @@ impl ObjectInfo {
return Ok(actual_size);
}
// Check if object is encrypted
// Managed SSE stores original size in x-rustfs-encryption-original-size metadata
// SSE-C stores original size in x-amz-server-side-encryption-customer-original-size
if let Some(size) = self.encryption_original_size()? {
return Ok(size);
}
@@ -839,19 +881,6 @@ mod tests {
assert!(!object.is_inline_fast_path_eligible(), "transitioned objects must fall back");
}
#[test]
fn minio_internal_encryption_metadata_is_not_treated_as_plaintext() {
let object = ObjectInfo {
user_defined: Arc::new(HashMap::from([(
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key".to_string(),
"sealed".to_string(),
)])),
..Default::default()
};
assert!(object.is_encrypted());
}
#[test]
fn versions_after_marker_handles_null_version_marker() {
let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
-17
View File
@@ -46,7 +46,6 @@ use crate::bucket::metadata_sys::BucketMetadataSys;
use crate::bucket::replication::{DynReplicationPool, ReplicationStats};
use crate::disk::DiskStore;
use crate::layout::endpoints::{EndpointServerPools, SetupType};
use crate::object_api::ObjectEncryptionResolver;
use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
@@ -160,8 +159,6 @@ pub struct InstanceContext {
/// workers (scanner/heal/tier/lifecycle) without touching another instance.
/// Replaces the process-global cancel-token static.
background_cancel_token: OnceLock<CancellationToken>,
/// Resolves object-encryption material at the application boundary.
object_encryption_resolver: OnceLock<Arc<dyn ObjectEncryptionResolver>>,
tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
#[cfg(test)]
@@ -200,7 +197,6 @@ impl InstanceContext {
local_disk_set_drives: Arc::new(RwLock::new(Vec::new())),
bucket_metadata_sys: std::sync::Mutex::new(None),
background_cancel_token: OnceLock::new(),
object_encryption_resolver: OnceLock::new(),
tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
#[cfg(test)]
@@ -213,19 +209,6 @@ impl InstanceContext {
self.lock_manager.clone()
}
/// Install the application-owned object-encryption resolver once.
pub fn set_object_encryption_resolver(
&self,
resolver: Arc<dyn ObjectEncryptionResolver>,
) -> Result<(), Arc<dyn ObjectEncryptionResolver>> {
self.object_encryption_resolver.set(resolver)
}
/// Return the configured object-encryption resolver, if startup installed one.
pub fn object_encryption_resolver(&self) -> Option<&dyn ObjectEncryptionResolver> {
self.object_encryption_resolver.get().map(Arc::as_ref)
}
/// Set this instance's S3 region.
///
/// Write-once: panics on a second write, preserving the startup fail-fast
+12 -111
View File
@@ -27,7 +27,7 @@ use crate::{
bucket::replication::{DynReplicationPool, ReplicationStats},
config::{get_global_storage_class, get_global_storage_class_snapshot, set_global_storage_class, storageclass},
disk::{DiskAPI, DiskOption, DiskStore, new_disk},
error::{Error, Result},
error::Result,
layout::endpoints::{EndpointServerPools, SetupType},
runtime::global::{
GLOBAL_BOOT_TIME, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
@@ -46,6 +46,7 @@ use crate::{
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_kms::{ObjectEncryptionService, get_global_encryption_service};
use rustfs_lock::client::LockClient;
use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
@@ -104,6 +105,10 @@ pub(crate) fn record_erasure_write_quorum_failure(stage: &'static str, dominant_
global_internode_metrics().record_erasure_write_quorum_failure(stage, dominant_error);
}
pub(crate) async fn object_encryption_service() -> Option<Arc<ObjectEncryptionService>> {
get_global_encryption_service().await
}
pub fn object_store_handle() -> Option<Arc<ECStore>> {
resolve_object_store_handle()
}
@@ -412,6 +417,10 @@ pub(crate) async fn clear_local_disk_id_map_for_test() {
local_disk_id_map_handle().write().await.clear();
}
pub(crate) async fn record_local_disk_id(instance_ctx: &Arc<InstanceContext>, disk_id: Uuid, endpoint: String) {
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
}
pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Option<Uuid>, endpoint: String) {
let id_map = local_disk_id_map_handle();
let mut disk_id_map = id_map.write().await;
@@ -427,53 +436,6 @@ pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Optio
}
}
pub(crate) async fn reconcile_local_disk_ids(
instance_ctx: &InstanceContext,
pool_endpoints: &[String],
selected: &[(Uuid, String)],
) {
let pool_endpoints = pool_endpoints.iter().map(String::as_str).collect::<HashSet<_>>();
let disk_id_map = instance_ctx.local_disk_id_map();
let mut disk_ids = disk_id_map.write().await;
disk_ids.retain(|_, registered_endpoint| !pool_endpoints.contains(registered_endpoint.as_str()));
disk_ids.extend(selected.iter().cloned());
}
pub(crate) async fn quarantine_local_disks(instance_ctx: &InstanceContext, endpoints: &[Endpoint]) -> Result<()> {
let slots = endpoints
.iter()
.map(|endpoint| {
Ok((
usize::try_from(endpoint.pool_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.set_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.disk_idx).map_err(|_| Error::CorruptedFormat)?,
))
})
.collect::<Result<Vec<_>>>()?;
let local_disk_map = instance_ctx.local_disk_map();
let mut local_disks = local_disk_map.write().await;
for endpoint in endpoints {
local_disks.insert(endpoint.to_string(), None);
}
drop(local_disks);
let set_drives = instance_ctx.local_disk_set_drives();
let mut local_set_drives = set_drives.write().await;
if local_set_drives.is_empty() {
return Ok(());
}
for (pool_idx, set_idx, disk_idx) in slots {
let disk = local_set_drives
.get_mut(pool_idx)
.and_then(|sets| sets.get_mut(set_idx))
.and_then(|disks| disks.get_mut(disk_idx))
.ok_or(Error::CorruptedFormat)?;
*disk = None;
}
Ok(())
}
pub(crate) async fn record_local_disks(instance_ctx: &Arc<InstanceContext>, disks: Vec<DiskStore>) {
let map = instance_ctx.local_disk_map();
let mut global_local_disk_map = map.write().await;
@@ -601,8 +563,8 @@ pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
#[cfg(test)]
mod tests {
use super::{
LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, reconcile_local_disk_ids,
replace_local_disk_id, set_local_node_name,
LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, replace_local_disk_id,
set_local_node_name,
};
use crate::disk::endpoint::Endpoint;
use rustfs_lock::{LocalClient, LockClient};
@@ -666,65 +628,4 @@ mod tests {
assert_eq!(local_disk_path_by_id(&disk_id).await, Some("endpoint-a".to_string()));
clear_local_disk_id_map_for_test().await;
}
#[tokio::test]
#[serial_test::serial]
async fn reconciling_pool_disk_ids_preserves_other_endpoints() {
let instance_ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let process_ctx = crate::runtime::global::current_ctx();
let bootstrap_ctx = crate::runtime::instance::bootstrap_ctx();
let retained_id = Uuid::new_v4();
let removed_id = Uuid::new_v4();
let selected_id = Uuid::new_v4();
let process_sentinel = Uuid::new_v4();
let bootstrap_sentinel = Uuid::new_v4();
instance_ctx.local_disk_id_map().write().await.extend([
(retained_id, "endpoint-a".to_string()),
(removed_id, "endpoint-b".to_string()),
]);
process_ctx
.local_disk_id_map()
.write()
.await
.insert(process_sentinel, "endpoint-b".to_string());
bootstrap_ctx
.local_disk_id_map()
.write()
.await
.insert(bootstrap_sentinel, "endpoint-b".to_string());
reconcile_local_disk_ids(
&instance_ctx,
&["endpoint-b".to_string(), "endpoint-c".to_string()],
&[(selected_id, "endpoint-c".to_string())],
)
.await;
let disk_ids = instance_ctx.local_disk_id_map();
let disk_ids = disk_ids.read().await;
assert_eq!(disk_ids.get(&retained_id).map(String::as_str), Some("endpoint-a"));
assert_eq!(disk_ids.get(&removed_id), None);
assert_eq!(disk_ids.get(&selected_id).map(String::as_str), Some("endpoint-c"));
drop(disk_ids);
assert_eq!(
process_ctx
.local_disk_id_map()
.read()
.await
.get(&process_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
assert_eq!(
bootstrap_ctx
.local_disk_id_map()
.read()
.await
.get(&bootstrap_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
process_ctx.local_disk_id_map().write().await.remove(&process_sentinel);
bootstrap_ctx.local_disk_id_map().write().await.remove(&bootstrap_sentinel);
}
}
+3 -1
View File
@@ -505,7 +505,9 @@ impl SetDisks {
}
fn file_info_has_encryption_metadata(meta: &FileInfo) -> bool {
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
meta.metadata
.keys()
.any(|name| http::is_encryption_metadata_key(name) || http::is_sse_header(name))
}
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
+7 -6
View File
@@ -147,17 +147,15 @@ use rustfs_object_capacity::capacity_scope::{
CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
};
use rustfs_s3_types::EventName;
#[cfg(test)]
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
use rustfs_utils::http::headers::{
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
};
use rustfs_utils::http::{
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_RESTORE_OPERATION_ID, contains_key_str, get_header_map, get_str, insert_str, is_object_encryption_marker,
remove_header_map,
SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, contains_key_str,
get_header_map, get_str, insert_str, is_encryption_metadata_key, remove_header_map,
};
use rustfs_utils::{
HashAlgorithm,
@@ -672,7 +670,10 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
}
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
metadata.keys().any(|key| is_object_encryption_marker(key))
metadata.keys().any(|key| is_encryption_metadata_key(key))
|| metadata.contains_key(SSEC_ALGORITHM_HEADER)
|| metadata.contains_key(SSEC_KEY_HEADER)
|| metadata.contains_key(SSEC_KEY_MD5_HEADER)
}
/// Per-set memoized capacity dirty scope.
+2 -71
View File
@@ -42,7 +42,6 @@ use crate::object_api::{GetObjectBodySource, get_object_body_cache_hook_suppress
use crate::services::tier::tier::{TierConfigMgr, TierOperationLease};
use crate::store::ECStore;
use futures::FutureExt as _;
use http::HeaderValue;
use std::future::Future;
fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Erasure> {
@@ -50,17 +49,6 @@ fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Er
.map_err(Error::from)
}
async fn get_object_reader_with_context(
ctx: &InstanceContext,
reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
range: Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
opts: &ObjectOptions,
headers: &HeaderMap<HeaderValue>,
) -> Result<(GetObjectReader, usize, i64)> {
GetObjectReader::new_with_resolver(reader, range, object_info, opts, headers, ctx.object_encryption_resolver()).await
}
/// Length of the full plaintext body when — and only when — this read's output
/// is exactly the object's complete plaintext, so the app-layer body cache may
/// serve it in place of the erasure read.
@@ -725,8 +713,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
size_bucket,
);
record_get_object_reader_path_observation(GET_OBJECT_PATH_CODEC_STREAMING, object_class, size_bucket);
let (mut reader, _offset, _length) =
get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
let (mut reader, _offset, _length) = GetObjectReader::new(stream, range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its
// now-redundant lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -758,8 +745,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
let (mut reader, offset, length) =
get_object_reader_with_context(&self.ctx, Box::new(rd), range, &object_info, opts, &h).await?;
let (mut reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its now-redundant
// lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -4550,61 +4536,6 @@ mod erasure_construction_tests {
}
}
#[cfg(test)]
mod object_encryption_resolver_wiring_tests {
use super::*;
use crate::object_api::{EncryptionResolutionError, ObjectEncryptionResolver, ReadEncryptionMaterial, ReadEncryptionRequest};
use std::io::Cursor;
use std::sync::atomic::{AtomicUsize, Ordering};
struct CountingResolver {
calls: AtomicUsize,
}
#[async_trait::async_trait]
impl ObjectEncryptionResolver for CountingResolver {
async fn resolve_read_material(
&self,
_request: ReadEncryptionRequest<'_>,
) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
self.calls.fetch_add(1, Ordering::Relaxed);
Ok(None)
}
}
#[tokio::test]
async fn get_object_reader_forwards_instance_resolver() {
let resolver = Arc::new(CountingResolver {
calls: AtomicUsize::new(0),
});
let ctx = InstanceContext::new();
assert!(
ctx.set_object_encryption_resolver(resolver.clone()).is_ok(),
"fresh context should accept resolver"
);
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: 1,
user_defined: Arc::new(HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())])),
..Default::default()
};
let result = get_object_reader_with_context(
&ctx,
Box::new(Cursor::new(Vec::<u8>::new())),
None,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await;
assert!(result.is_err(), "resolver returning no material must fail closed");
assert_eq!(resolver.calls.load(Ordering::Relaxed), 1);
}
}
#[cfg(test)]
pub(in crate::set_disk::ops) mod hermetic_set_disks_support {
//! Shared hermetic `SetDisks` construction for the ops tests below: the
+7 -7
View File
@@ -199,7 +199,7 @@ impl SetDisks {
);
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read_version_optimized(
&self,
bucket: &str,
@@ -238,7 +238,7 @@ impl SetDisks {
}
#[tracing::instrument(level = "debug", skip(self))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_fileinfo(
&self,
bucket: &str,
@@ -410,7 +410,7 @@ impl SetDisks {
Ok((fi, parts_metadata, op_online_disks))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_info_and_quorum(
&self,
bucket: &str,
@@ -605,7 +605,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_with_fileinfo<W>(
// &self,
bucket: &str,
@@ -1140,7 +1140,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_decode_reader_with_fileinfo(
bucket: &str,
object: &str,
@@ -1296,7 +1296,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn build_codec_streaming_part_reader(
bucket: &str,
object: &str,
@@ -1469,7 +1469,7 @@ fn multipart_part_checksum_algo(fi: &FileInfo, part_number: usize) -> HashAlgori
/// `get_object_with_fileinfo` (backlog#870) so both report the same
/// stage-duration semantics.
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn setup_multipart_part_readers(
files: &[FileInfo],
disks: &[Option<DiskStore>],
+1 -21
View File
@@ -313,8 +313,7 @@ impl ECStore {
let mut times = 0;
let mut interval = 1;
loop {
match init_format::connect_load_init_formats_with_instance_ctx(
&instance_ctx,
match init_format::connect_load_init_formats(
pool_first_is_local,
&mut disks,
pool_eps.set_count,
@@ -1260,16 +1259,6 @@ mod tests {
let registered: Vec<String> = instance_ctx.local_disk_map().read().await.keys().cloned().collect();
assert_eq!(registered.len(), 4, "the passed context must register all four local disks");
let registered_disk_ids = instance_ctx.local_disk_id_map();
let registered_disk_ids = registered_disk_ids.read().await;
assert_eq!(registered_disk_ids.len(), 4, "the passed context must publish all four disk IDs");
for endpoint in registered_disk_ids.values() {
assert!(
registered.contains(endpoint),
"every disk ID in the passed context must resolve to one of its registered endpoints"
);
}
drop(registered_disk_ids);
let bootstrap = crate::runtime::instance::bootstrap_ctx();
assert_ne!(
bootstrap.deployment_id(),
@@ -1284,15 +1273,6 @@ mod tests {
"the bootstrap context must not absorb the fresh store's disks"
);
}
drop(bootstrap_map);
let bootstrap_disk_ids = bootstrap.local_disk_id_map();
let bootstrap_disk_ids = bootstrap_disk_ids.read().await;
for endpoint in bootstrap_disk_ids.values() {
assert!(
!registered.contains(endpoint),
"the bootstrap context must not absorb the fresh store's disk IDs"
);
}
}
#[tokio::test]
+58 -227
View File
@@ -16,8 +16,6 @@ use crate::config::storageclass;
use crate::disk::error_reduce::{count_errs, reduce_write_quorum_errs};
use crate::disk::{self, DiskAPI};
use crate::error::{Error, Result};
use crate::runtime::instance::InstanceContext;
use crate::runtime::sources::{quarantine_local_disks, reconcile_local_disk_ids};
use crate::{
disk::{
DiskInfoOptions, DiskOption, DiskStore, FORMAT_CONFIG_FILE, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET,
@@ -28,10 +26,7 @@ use crate::{
layout::endpoints::Endpoints,
};
use futures::{future::join_all, stream, stream::StreamExt};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
use std::collections::{HashMap, HashSet};
use tokio::io::AsyncReadExt;
use tracing::{debug, error, info, warn};
use uuid::Uuid;
@@ -67,25 +62,12 @@ pub async fn init_disks(eps: &Endpoints, opt: &DiskOption) -> (Vec<Option<DiskSt
(res, errors)
}
#[cfg(test)]
pub async fn connect_load_init_formats(
first_disk: bool,
disks: &mut [Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
) -> Result<FormatV3> {
let instance_ctx = crate::runtime::global::current_ctx();
connect_load_init_formats_with_instance_ctx(&instance_ctx, first_disk, disks, set_count, set_drive_count, deployment_id).await
}
pub(crate) async fn connect_load_init_formats_with_instance_ctx(
instance_ctx: &Arc<InstanceContext>,
first_disk: bool,
disks: &mut [Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
) -> Result<FormatV3> {
let (formats, errs) = load_format_erasure_all(disks, false).await;
@@ -116,7 +98,7 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
match try_migrate_format(disks, &formats, set_count, set_drive_count).await {
Ok(LegacyFormatOutcome::Migrated { format, quorum_members }) => {
info!("Migrated format from MinIO config");
retain_format_quorum_members(instance_ctx, disks, &format, &quorum_members, set_drive_count).await?;
retain_format_quorum_members(disks, &format, &quorum_members, set_drive_count).await?;
return Ok(*format);
}
Ok(LegacyFormatOutcome::Incompatible) => {
@@ -133,7 +115,7 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
Err(e) => return Err(e),
}
if all_unformatted {
let fm = init_format_erasure(instance_ctx, disks, set_count, set_drive_count, deployment_id).await?;
let fm = init_format_erasure(disks, set_count, set_drive_count, deployment_id).await?;
return Ok(fm);
}
}
@@ -158,13 +140,12 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
None => select_format_erasure_in_quorum(&formats, 0)?,
};
check_format_erasure_value_for_topology(&fm, formats.len(), set_drive_count)?;
retain_format_quorum_members(instance_ctx, disks, &fm, &quorum_members, set_drive_count).await?;
retain_format_quorum_members(disks, &fm, &quorum_members, set_drive_count).await?;
Ok(fm)
}
async fn retain_format_quorum_members(
instance_ctx: &Arc<InstanceContext>,
disks: &mut [Option<DiskStore>],
format: &FormatV3,
quorum_members: &[bool],
@@ -173,73 +154,26 @@ async fn retain_format_quorum_members(
if set_drive_count == 0 || quorum_members.len() != disks.len() {
return Err(Error::CorruptedFormat);
}
let pool_idx = disks
.iter()
.flatten()
.map(|disk| disk.endpoint().pool_idx)
.find_map(|pool_idx| usize::try_from(pool_idx).ok());
let registered_endpoints = if let Some(pool_idx) = pool_idx {
instance_ctx
.local_disk_set_drives()
.read()
.await
.get(pool_idx)
.map(|sets| {
sets.iter()
.flat_map(|set| set.iter())
.map(|disk| disk.as_ref().map(|disk| disk.endpoint()))
.collect::<Vec<_>>()
})
.filter(|endpoints| endpoints.len() == disks.len())
} else {
None
};
let endpoints = disks
.iter()
.enumerate()
.map(|(index, disk)| {
disk.as_ref()
.map(|disk| disk.endpoint())
.or_else(|| registered_endpoints.as_ref()?.get(index)?.clone())
})
.collect::<Vec<_>>();
let mut member_disk_ids = vec![None; disks.len()];
let mut local_pool_endpoints = Vec::new();
let mut selected_local_disk_ids = Vec::new();
let mut quarantined_endpoints = Vec::new();
for (index, ((disk, endpoint), belongs_to_quorum)) in disks.iter().zip(&endpoints).zip(quorum_members).enumerate() {
if let Some(endpoint) = endpoint.as_ref().filter(|endpoint| endpoint.is_local) {
local_pool_endpoints.push(endpoint.to_string());
if !belongs_to_quorum {
quarantined_endpoints.push(endpoint.clone());
}
for (disk, belongs_to_quorum) in disks.iter().zip(quorum_members) {
if !belongs_to_quorum && let Some(disk) = disk {
disk.set_disk_id(None).await?;
}
}
for (index, (disk, belongs_to_quorum)) in disks.iter().zip(quorum_members).enumerate() {
if !belongs_to_quorum {
continue;
}
let disk = disk.as_ref().ok_or(Error::CorruptedFormat)?;
let disk_id = format
.erasure
.sets
.get(index / set_drive_count)
.and_then(|set| set.get(index % set_drive_count))
.copied()
.ok_or(Error::CorruptedFormat)?;
member_disk_ids[index] = Some(disk_id);
if disk.is_local() {
selected_local_disk_ids.push((disk_id, disk.endpoint().to_string()));
}
disk.as_ref()
.ok_or(Error::CorruptedFormat)?
.set_disk_id(Some(*disk_id))
.await?;
}
quarantine_local_disks(instance_ctx, &quarantined_endpoints).await?;
for (disk, disk_id) in disks.iter().zip(member_disk_ids) {
if let Some(disk) = disk {
disk.set_disk_id_state(disk_id).await?;
}
}
reconcile_local_disk_ids(instance_ctx, &local_pool_endpoints, &selected_local_disk_ids).await;
for (disk, belongs_to_quorum) in disks.iter_mut().zip(quorum_members) {
if !belongs_to_quorum {
*disk = None;
@@ -273,8 +207,7 @@ pub fn check_disk_fatal_errs(errs: &[Option<DiskError>]) -> disk::error::Result<
}
async fn init_format_erasure(
instance_ctx: &Arc<InstanceContext>,
disks: &mut [Option<DiskStore>],
disks: &[Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
@@ -294,11 +227,9 @@ async fn init_format_erasure(
}
}
write_format_file_all(disks, &fms).await?;
let format = get_format_erasure_in_quorum(&fms, 0)?;
let quorum_members = vec![true; disks.len()];
retain_format_quorum_members(instance_ctx, disks, &format, &quorum_members, set_drive_count).await?;
Ok(format)
save_format_file_all(disks, &fms).await?;
get_format_erasure_in_quorum(&fms, 0)
}
/// Outcome of attempting to migrate an on-disk MinIO `format.json`.
@@ -417,59 +348,44 @@ async fn try_migrate_format(
.iter()
.zip(&formats_to_write)
.zip(rustfs_formats)
.filter_map(|((disk, format), existing)| {
if existing.is_some() {
return None;
}
Some(write_format_file(disk.as_ref()?, format.as_ref()?))
});
.filter(|&((disk, _), existing)| disk.is_some() && existing.is_none())
.map(|((disk, format), _)| save_format_file(disk, format));
let mut write_error = None;
for result in join_all(writes).await {
if let Err(error) = result {
write_error.get_or_insert(error);
}
}
let (persisted_formats, persisted_errors) = load_format_erasure_all(disks, false).await;
let Some((persisted_format, quorum_members)) =
select_persisted_migration_format(&persisted_formats, persisted_errors, &format, set_drive_count, write_error)?
else {
return Ok(LegacyFormatOutcome::Incompatible);
};
Ok(LegacyFormatOutcome::Migrated {
format: Box::new(persisted_format),
quorum_members,
})
}
fn select_persisted_migration_format(
persisted_formats: &[Option<FormatV3>],
persisted_errors: Vec<Option<DiskError>>,
reference: &FormatV3,
set_drive_count: usize,
write_error: Option<DiskError>,
) -> Result<Option<(FormatV3, Vec<bool>)>> {
if !formats_match_reference_slots(persisted_formats, reference, 0) {
return Ok(None);
for disk in disks.iter().flatten() {
disk.set_disk_id(None).await?;
}
let quorum_error = match select_format_erasure_in_quorum(persisted_formats, 0) {
Ok((format, quorum_members)) => {
check_format_erasure_value_for_topology(&format, persisted_formats.len(), set_drive_count)?;
return Ok(Some((format, quorum_members)));
}
Err(error) => error,
};
let (persisted_formats, persisted_errors) = load_format_erasure_all(disks, false).await;
if let Some(error) = persisted_errors
.into_iter()
.flatten()
.find(|error| !matches!(error, DiskError::UnformattedDisk | DiskError::DiskNotFound))
{
return match error {
DiskError::CorruptedFormat | DiskError::CorruptedBackend | DiskError::InconsistentDisk => Ok(None),
DiskError::CorruptedFormat | DiskError::CorruptedBackend | DiskError::InconsistentDisk => {
Ok(LegacyFormatOutcome::Incompatible)
}
error => Err(error.into()),
};
}
Err(write_error.map_or(quorum_error, Into::into))
if !formats_match_reference_slots(&persisted_formats, &format, 0) {
return Ok(LegacyFormatOutcome::Incompatible);
}
let (persisted_format, quorum_members) = match select_format_erasure_in_quorum(&persisted_formats, 0) {
Ok(selected) => selected,
Err(error) => return Err(write_error.map_or(error, Into::into)),
};
check_format_erasure_value_for_topology(&persisted_format, disks.len(), set_drive_count)?;
Ok(LegacyFormatOutcome::Migrated {
format: Box::new(persisted_format),
quorum_members,
})
}
fn legacy_format_max_bytes(disk_count: usize) -> Result<usize> {
@@ -760,12 +676,13 @@ pub async fn load_format_erasure(disk: &DiskStore, heal: bool) -> disk::error::R
Ok(fm)
}
async fn write_format_file_all(disks: &[Option<DiskStore>], formats: &[Option<FormatV3>]) -> disk::error::Result<()> {
let futures = disks.iter().zip(formats).map(|(disk, format)| async move {
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
let format = format.as_ref().ok_or_else(|| DiskError::other("format is none"))?;
write_format_file(disk, format).await
});
async fn save_format_file_all(disks: &[Option<DiskStore>], formats: &[Option<FormatV3>]) -> disk::error::Result<()> {
let mut futures = Vec::with_capacity(disks.len());
for (i, disk) in disks.iter().enumerate() {
futures.push(save_format_file(disk, &formats[i]));
}
let mut errors = Vec::with_capacity(disks.len());
let results = join_all(futures).await;
@@ -796,13 +713,6 @@ pub async fn save_format_file(disk: &Option<DiskStore>, format: &Option<FormatV3
return Err(DiskError::other("format is none"));
};
write_format_file(disk, format).await?;
disk.set_disk_id(Some(format.erasure.this)).await?;
Ok(())
}
async fn write_format_file(disk: &DiskStore, format: &FormatV3) -> disk::error::Result<()> {
let json_data = format.to_json()?;
let tmpfile = Uuid::new_v4().to_string();
@@ -813,6 +723,8 @@ async fn write_format_file(disk: &DiskStore, format: &FormatV3) -> disk::error::
disk.rename_file(RUSTFS_META_BUCKET, tmpfile.as_str(), RUSTFS_META_BUCKET, FORMAT_CONFIG_FILE)
.await?;
disk.set_disk_id(Some(format.erasure.this)).await?;
Ok(())
}
@@ -826,9 +738,7 @@ pub fn ec_drives_no_config(set_drive_count: usize) -> Result<usize> {
mod tests {
use super::*;
use crate::layout::endpoint::Endpoint;
use crate::runtime::global::{current_ctx, reset_local_disk_test_state};
use crate::runtime::sources::{local_disk_path_by_id, record_local_disks};
use crate::store::peer::{find_local_disk_by_ref, prewarm_local_disk_id_map_with_instance_ctx};
use crate::runtime::sources::{clear_local_disk_id_map_for_test, local_disk_path_by_id};
use serial_test::serial;
async fn local_disks(count: usize) -> (tempfile::TempDir, Vec<Option<DiskStore>>) {
@@ -1135,7 +1045,7 @@ mod tests {
#[tokio::test]
#[serial]
async fn existing_format_load_publishes_only_validated_quorum_disk_ids() {
reset_local_disk_test_state().await;
clear_local_disk_id_map_for_test().await;
let (_temp_dir, mut disks) = local_disks(5).await;
let canonical = FormatV3::new(1, 5);
for (index, disk) in disks.iter().enumerate().take(3) {
@@ -1145,56 +1055,19 @@ mod tests {
.await
.expect("canonical format should be written");
}
let canonical_id = canonical.erasure.sets[0][0];
let mut wrong_slot = canonical.clone();
wrong_slot.erasure.this = wrong_slot.erasure.sets[0][0];
save_format_file(&disks[3], &Some(wrong_slot))
.await
.expect("wrong-slot format should be written");
let mut foreign = FormatV3::new(1, 5);
foreign.erasure.this = foreign.erasure.sets[0][4];
let foreign_id = foreign.erasure.this;
save_format_file(&disks[4], &Some(foreign))
.await
.expect("foreign format should be written");
let mut collision = FormatV3::new(1, 5);
collision.erasure.sets[0][3] = canonical_id;
collision.erasure.this = canonical_id;
save_format_file(&disks[3], &Some(collision))
.await
.expect("foreign collision format should be written");
let canonical_id = canonical.erasure.sets[0][0];
let canonical_endpoint = disks[0].as_ref().expect("canonical disk should exist").endpoint().to_string();
let collision_endpoint = disks[3].as_ref().expect("collision disk should exist").endpoint().to_string();
let foreign_endpoint = disks[4].as_ref().expect("foreign disk should exist").endpoint().to_string();
let prewarm_endpoints = Endpoints::from(
disks
.iter()
.map(|disk| disk.as_ref().expect("test disk should exist").endpoint())
.collect::<Vec<_>>(),
);
for disk in disks.iter().flatten() {
disk.set_disk_id(None).await.expect("test disk ID should be reset");
}
let (prewarm_disks, prewarm_errors) = init_disks(
&prewarm_endpoints,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await;
assert!(prewarm_errors.iter().all(Option::is_none));
let prewarm_disks = prewarm_disks
.into_iter()
.map(|disk| disk.expect("prewarm disk should exist"))
.collect::<Vec<_>>();
let instance_ctx = current_ctx();
record_local_disks(&instance_ctx, prewarm_disks.clone()).await;
*instance_ctx.local_disk_set_drives().write().await = vec![vec![prewarm_disks.into_iter().map(Some).collect()]];
prewarm_local_disk_id_map_with_instance_ctx(&instance_ctx).await;
instance_ctx
.local_disk_id_map()
.write()
.await
.insert(canonical_id, collision_endpoint.clone());
assert!(local_disk_path_by_id(&foreign_id).await.is_some(), "foreign ID should be prewarmed");
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(collision_endpoint));
disks[4] = None;
connect_load_init_formats(true, &mut disks, 1, 5, None)
.await
@@ -1202,51 +1075,9 @@ mod tests {
assert!(disks[..3].iter().all(Option::is_some));
assert!(disks[3..].iter().all(Option::is_none));
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(canonical_endpoint.clone()));
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(canonical_endpoint));
assert_eq!(local_disk_path_by_id(&foreign_id).await, None);
assert!(
instance_ctx
.local_disk_map()
.read()
.await
.get(&foreign_endpoint)
.is_some_and(Option::is_none)
);
assert!(instance_ctx.local_disk_set_drives().read().await[0][0][4].is_none());
assert!(find_local_disk_by_ref(&foreign_id.to_string()).await.is_none());
assert_eq!(
find_local_disk_by_ref(&canonical_id.to_string())
.await
.map(|disk| disk.endpoint().to_string()),
Some(canonical_endpoint)
);
reset_local_disk_test_state().await;
}
#[test]
fn persisted_migration_quorum_ignores_nonmember_read_errors() {
for drive_count in [3, 4] {
let reference = FormatV3::new(1, drive_count);
let required = drive_count / 2 + 1;
let mut formats = vec![None; drive_count];
let mut errors = (0..drive_count).map(|_| None).collect::<Vec<Option<DiskError>>>();
for (index, format) in formats.iter_mut().enumerate().take(required) {
let mut disk_format = reference.clone();
disk_format.erasure.this = reference.erasure.sets[0][index];
*format = Some(disk_format);
}
errors[drive_count - 1] = Some(DiskError::FaultyDisk);
let (selected, members) =
select_persisted_migration_format(&formats, errors, &reference, drive_count, Some(DiskError::FaultyDisk))
.expect("a persisted strict majority must outrank a nonmember read error")
.expect("the persisted formats should be compatible");
assert_eq!(selected.shared_identity(), reference.shared_identity());
assert_eq!(members.iter().filter(|member| **member).count(), required);
assert!(members[..required].iter().all(|member| *member));
assert!(members[required..].iter().all(|member| !*member));
}
clear_local_disk_id_map_for_test().await;
}
#[tokio::test]
+1 -1
View File
@@ -238,7 +238,7 @@ impl ECStore {
}
#[instrument(skip(self, data))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_put_object_part(
&self,
bucket: &str,
+2 -2
View File
@@ -815,7 +815,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_get_object_reader(
&self,
bucket: &str,
@@ -849,7 +849,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_put_object(
&self,
bucket: &str,
+2 -73
View File
@@ -28,26 +28,8 @@ async fn remember_local_disk_id(disk: &DiskStore) -> Option<Uuid> {
async fn remember_local_disk_id_with_instance_ctx(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore) -> Option<Uuid> {
let disk_id = disk.get_disk_id().await.ok().flatten()?;
record_local_disk_id_if_active(instance_ctx, disk, disk_id)
.await
.then_some(disk_id)
}
async fn record_local_disk_id_if_active(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore, disk_id: Uuid) -> bool {
let endpoint = disk.endpoint().to_string();
let local_disk_map = instance_ctx.local_disk_map();
let local_disks = local_disk_map.read().await;
let Some(active_disk) = local_disks.get(&endpoint).and_then(Option::as_ref) else {
return false;
};
if !Arc::ptr_eq(active_disk, disk) {
return false;
}
// Lock order is local_disk_map -> local_disk_id_map so quarantine is the
// linearization point for rejecting an in-flight stale disk snapshot.
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
true
runtime_sources::record_local_disk_id(instance_ctx, disk_id, disk.endpoint().to_string()).await;
Some(disk_id)
}
pub async fn find_local_disk(disk_path: &str) -> Option<DiskStore> {
@@ -246,7 +228,6 @@ pub async fn get_disk_infos(disks: &[Option<DiskStore>]) -> Vec<Option<DiskInfo>
#[cfg(test)]
mod tests {
use super::*;
use crate::disk::new_disk;
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
fn single_local_disk_pools(dir: &std::path::Path) -> EndpointServerPools {
@@ -333,56 +314,4 @@ mod tests {
);
}
}
#[tokio::test]
async fn stale_local_disk_snapshot_cannot_repopulate_the_id_registry() {
let temp_dir = tempfile::tempdir().expect("create temp disk dir");
let endpoint_pools = single_local_disk_pools(temp_dir.path());
let instance_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools)
.await
.expect("local disk should be registered");
let disk = instance_ctx
.local_disk_map()
.read()
.await
.values()
.find_map(|disk| disk.clone())
.expect("registered local disk");
let endpoint = disk.endpoint().to_string();
let disk_id = Uuid::new_v4();
let local_disk_map = instance_ctx.local_disk_map();
let mut quarantine = local_disk_map.write().await;
let replacement = new_disk(
&disk.endpoint(),
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("replacement disk should initialize");
assert!(!Arc::ptr_eq(&disk, &replacement));
let task_ctx = instance_ctx.clone();
let task_disk = disk.clone();
let remember = tokio::spawn(async move { record_local_disk_id_if_active(&task_ctx, &task_disk, disk_id).await });
tokio::task::yield_now().await;
quarantine.insert(endpoint.clone(), Some(replacement.clone()));
drop(quarantine);
assert!(!remember.await.expect("stale lookup task should complete"));
assert!(!instance_ctx.local_disk_id_map().read().await.contains_key(&disk_id));
let active = instance_ctx
.local_disk_map()
.read()
.await
.get(&endpoint)
.cloned()
.flatten()
.expect("replacement disk should remain registered");
assert!(Arc::ptr_eq(&active, &replacement));
assert!(record_local_disk_id_if_active(&instance_ctx, &replacement, disk_id).await);
assert_eq!(instance_ctx.local_disk_id_map().read().await.get(&disk_id), Some(&endpoint));
}
}
+2 -2
View File
@@ -2,7 +2,7 @@
## MinIO-generated encrypted fixtures
`rustfs/src/storage/minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
`minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
`.\rustfs\scripts\minio_fixture_lab\lab.py`.
It currently covers multipart fixtures for:
@@ -20,5 +20,5 @@ Example:
```powershell
$env:RUSTFS_MINIO_FIXTURE_ROOT = '.\rustfs\tmp\minio-fixture-lab-local-key'
$env:RUSTFS_MINIO_STATIC_KMS_KEY_B64 = '<base64-32-byte-local-minio-kms-key>'
cargo +1.97.1 test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored
cargo +1.97.1 test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
```
@@ -4,13 +4,14 @@ use std::fs;
use std::io::Cursor;
use std::path::{Path, PathBuf};
use super::sse::SseObjectEncryptionResolver;
use super::storage_api::ecstore_test_support::{
DiskAPI as _, DiskOption, Endpoint, Erasure, GetObjectReader, ObjectInfo, ObjectOptions, create_bitrot_reader, new_disk,
};
mod storage_api;
use rustfs_filemeta::{FileInfo, FileInfoOpts, get_file_info};
use serde::Deserialize;
use sha2::{Digest, Sha256};
use storage_api::minio_generated_read::{
DiskAPI as _, DiskOption, Endpoint, Erasure, GetObjectReader, ObjectInfo, ObjectOptions, create_bitrot_reader, new_disk,
};
use temp_env::async_with_vars;
use tokio::io::{AsyncReadExt, AsyncWrite};
@@ -48,7 +49,7 @@ impl AsyncWrite for VecAsyncWriter {
fn fixture_root() -> PathBuf {
std::env::var_os("RUSTFS_MINIO_FIXTURE_ROOT")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../crates/rio-v2/tests/fixtures/minio-generated"))
.unwrap_or_else(|| PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../rio-v2/tests/fixtures/minio-generated"))
}
fn case_dir(case_id: &str) -> PathBuf {
@@ -136,14 +137,12 @@ async fn read_fixture_plaintext(encrypted: Vec<u8>, object_info: ObjectInfo, kms
("RUSTFS_SSE_S3_MASTER_KEY", None::<String>),
],
async move {
let resolver = SseObjectEncryptionResolver;
let (mut reader, offset, length) = GetObjectReader::new_with_resolver(
let (mut reader, offset, length) = GetObjectReader::new(
Box::new(Cursor::new(encrypted)),
None,
&object_info,
&ObjectOptions::default(),
&http::HeaderMap::new(),
Some(&resolver),
)
.await
.map_err(|err| format!("construct GetObjectReader from MinIO raw fixture: {err:?}"))?;
@@ -220,12 +219,11 @@ async fn encrypted_fixture_bytes(case_dir: &Path, manifest: &ManifestRecord, fil
readers.push(reader);
}
let erasure = Erasure::try_new(
let erasure = Erasure::new(
file_info.erasure.data_blocks,
file_info.erasure.parity_blocks,
file_info.erasure.block_size,
)
.expect("fixture erasure geometry");
);
let mut writer = VecAsyncWriter::default();
let (written, err) = erasure.decode(&mut writer, readers, 0, part.size, part.size).await;
if let Some(err) = err {
-7
View File
@@ -27,14 +27,7 @@ categories = ["web-programming", "development-tools"]
[lib]
doctest = false
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
thiserror.workspace = true
+2 -4
View File
@@ -27,12 +27,10 @@ documentation = "https://docs.rs/rustfs-filemeta/latest/rustfs_filemeta/"
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-utils/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-utils/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-utils/hotpath-cpu"]
hotpath = ["dep:hotpath", "hotpath/hotpath"]
[dependencies]
hotpath.workspace = true
hotpath = { workspace = true, optional = true }
crc-fast = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
+3 -3
View File
@@ -19,7 +19,7 @@ impl FileMeta {
!matches!(Self::check_xl2_v1(buf), Err(_e))
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn load(buf: &[u8]) -> Result<FileMeta> {
let mut xl = FileMeta::default();
xl.unmarshal_msg(buf)?;
@@ -112,7 +112,7 @@ impl FileMeta {
Ok((bin_len, remaining))
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result<u64> {
let i = buf.len() as u64;
@@ -326,7 +326,7 @@ impl FileMeta {
}
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
let mut wr = Vec::new();
-41
View File
@@ -29,48 +29,7 @@ categories = ["web-programming", "development-tools", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-common/hotpath",
"rustfs-concurrency/hotpath",
"rustfs-config/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-madmin/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-utils/hotpath",
"rustfs-test-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-concurrency/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-test-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-concurrency/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-test-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true }
rustfs-concurrency = { workspace = true }
rustfs-ecstore = { workspace = true }
-1
View File
@@ -159,7 +159,6 @@ impl ErasureSetHealer {
/// execute erasure set heal with resume
#[tracing::instrument(skip(self, buckets), fields(set_disk_id = %set_disk_id, bucket_count = buckets.len()))]
#[hotpath::measure]
pub async fn heal_erasure_set(&self, buckets: &[String], set_disk_id: &str) -> Result<()> {
debug!(
target: "rustfs::heal::erasure_healer",
-3
View File
@@ -584,7 +584,6 @@ impl HealTask {
}
#[tracing::instrument(skip(self), fields(task_id = %self.id, heal_type = ?self.heal_type))]
#[hotpath::measure]
pub async fn execute(&self) -> Result<()> {
// update status and timestamps atomically to avoid race conditions
let now = SystemTime::now();
@@ -760,7 +759,6 @@ impl HealTask {
// specific heal implementation method
#[tracing::instrument(skip(self), fields(bucket = %bucket, object = %object, version_id = ?version_id))]
#[hotpath::measure]
async fn heal_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
debug!(
target: "rustfs::heal::task",
@@ -1406,7 +1404,6 @@ impl HealTask {
self.heal_bucket_objects(bucket, prefix).await
}
#[hotpath::measure]
async fn heal_bucket_objects(&self, bucket: &str, prefix: &str) -> Result<()> {
let mut continuation_token: Option<String> = None;
let mut scanned = 0u64;
-48
View File
@@ -28,55 +28,7 @@ documentation = "https://docs.rs/rustfs-iam/latest/rustfs_iam/"
[lints]
workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-crypto/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-io-metrics/hotpath",
"rustfs-madmin/hotpath",
"rustfs-policy/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-utils/hotpath",
"rustfs-test-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-crypto/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-test-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-crypto/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-test-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-credentials = { workspace = true }
rustfs-config = { workspace = true, features = ["server-config-model"] }
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
+27 -58
View File
@@ -1139,11 +1139,8 @@ impl OidcSys {
let mut policies = Vec::new();
let mut groups = Vec::new();
// Role-policy and claim-based authorization are separate OIDC modes. When a
// role policy is configured, group claims still provide group context but
// must not also become policy names.
let has_role_policy = !config.role_policy.trim().is_empty();
if has_role_policy {
// Add default role policy if configured
if !config.role_policy.is_empty() {
for policy in config.role_policy.split(',') {
let policy = policy.trim();
if !policy.is_empty() {
@@ -1152,20 +1149,21 @@ impl OidcSys {
}
}
// Map groups claim to policies
for group in &claims.groups {
groups.push(group.clone());
if !has_role_policy {
let policy_name = if config.claim_prefix.is_empty() {
group.clone()
} else {
format!("{}{}", config.claim_prefix, group)
};
policies.push(policy_name);
}
let policy_name = if config.claim_prefix.is_empty() {
group.clone()
} else {
format!("{}{}", config.claim_prefix, group)
};
policies.push(policy_name);
}
if !has_role_policy && config.claim_name != config.groups_claim {
for val in extract_groups_claim(&claims.raw, &config.claim_name) {
// Map primary claim (if different from groups)
if config.claim_name != config.groups_claim {
let claim_values = extract_groups_claim(&claims.raw, &config.claim_name);
for val in claim_values {
let policy_name = if config.claim_prefix.is_empty() {
val
} else {
@@ -3203,22 +3201,26 @@ mod tests {
}
#[test]
fn role_policy_does_not_map_groups_as_policies() {
let mut config = test_config("authentik");
config.role_policy = "consoleAdmin".to_string();
config.claim_name = "policy".to_string();
fn test_map_claims_to_policies_with_provider() {
let mut config = test_config("okta");
config.role_policy = "readwrite".to_string();
config.display_name = "Okta".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["authentik Admins".to_string(), "users".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
sub: "user123".to_string(),
email: "user@example.com".to_string(),
username: "user".to_string(),
groups: vec!["admin".to_string(), "devs".to_string()],
raw: HashMap::new(),
};
let (policies, groups) = sys.map_claims_to_policies("authentik", &claims);
assert_eq!(groups, vec!["authentik Admins", "users"]);
assert_eq!(policies, vec!["consoleAdmin"]);
let (policies, groups) = sys.map_claims_to_policies("okta", &claims);
assert_eq!(groups, vec!["admin", "devs"]);
assert!(policies.contains(&"readwrite".to_string()));
assert!(policies.contains(&"admin".to_string()));
assert!(policies.contains(&"devs".to_string()));
}
#[test]
@@ -3243,39 +3245,6 @@ mod tests {
assert_eq!(policies.len(), 1);
}
#[test]
fn blank_role_policy_uses_claim_mapping() {
let mut config = test_config("keycloak");
config.role_policy = " ".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["readonly".to_string()],
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["readonly"]);
assert_eq!(policies, vec!["readonly"]);
}
#[test]
fn claim_mapping_keeps_groups_with_distinct_primary_claim() {
let mut config = test_config("keycloak");
config.claim_name = "policy".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["developers".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["developers"]);
assert_eq!(policies, vec!["developers", "readonly"]);
}
#[test]
fn test_list_providers() {
let mut config = test_config("keycloak");
-2
View File
@@ -469,7 +469,6 @@ impl ObjectStore {
});
}
#[hotpath::measure]
async fn list_all_iamconfig_items(&self) -> Result<HashMap<String, Vec<String>>> {
let (tx, mut rx) = mpsc::channel::<StringOrErr>(100);
@@ -509,7 +508,6 @@ impl ObjectStore {
Ok(res)
}
#[hotpath::measure]
async fn load_policy_doc_concurrent(&self, names: &[String], mode: LoadMode) -> Result<Vec<PolicyDoc>> {
let mut futures = Vec::with_capacity(names.len());
-7
View File
@@ -27,14 +27,7 @@ categories = ["development-tools", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-metrics/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-metrics/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-metrics/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["io-util", "fs", "sync", "rt-multi-thread"] }
-27
View File
@@ -28,36 +28,9 @@ categories = ["development-tools", "filesystem"]
name = "metrics_pipeline"
harness = false
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"rustfs-common/hotpath",
"rustfs-s3-ops/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-s3-ops/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-s3-ops/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
metrics = { workspace = true }
rustfs-common = { workspace = true }
rustfs-s3-ops = { workspace = true }
rustfs-utils = { workspace = true, features = ["ip"] }
num_cpus = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
+54 -199
View File
@@ -15,7 +15,7 @@
use metrics::{counter, gauge};
use std::collections::HashMap;
use std::sync::{
Arc, LazyLock, OnceLock, RwLock,
Arc, LazyLock, RwLock,
atomic::{AtomicU64, Ordering},
};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
@@ -40,7 +40,6 @@ pub const INTERNODE_MSGPACK_CODEC_JSON: &str = "json";
const OPERATION_LABEL: &str = "operation";
const BACKEND_LABEL: &str = "backend";
const SERVER_LABEL: &str = "server";
const CLASSIFICATION_LABEL: &str = "classification";
const STAGE_LABEL: &str = "stage";
const DOMINANT_ERROR_LABEL: &str = "dominant_error";
@@ -78,93 +77,74 @@ pub struct InternodeOperationMetricDescriptor {
pub labels: &'static [&'static str],
}
const SERVER_OPERATION_BACKEND_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL];
const SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS: &[&str] =
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL];
const SERVER_OPERATION_BACKEND_HTTP_VERSION_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL];
const SERVER_QUORUM_FAILURE_LABELS: &[&str] = &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL];
const OPERATION_BACKEND_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL];
const OPERATION_BACKEND_CLASSIFICATION_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL];
const OPERATION_BACKEND_HTTP_VERSION_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL];
const QUORUM_FAILURE_LABELS: &[&str] = &[STAGE_LABEL, DOMINANT_ERROR_LABEL];
pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &[
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_SENT_BYTES_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_RECV_BYTES_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_ERRORS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_DURATION_MS,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_RETRIES_TOTAL,
labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL,
labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_HTTP_VERSIONS_TOTAL,
labels: SERVER_OPERATION_BACKEND_HTTP_VERSION_LABELS,
labels: OPERATION_BACKEND_HTTP_VERSION_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: ERASURE_WRITE_QUORUM_FAILURES_TOTAL,
labels: SERVER_QUORUM_FAILURE_LABELS,
labels: QUORUM_FAILURE_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_PAYLOAD_BYTES,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
InternodeOperationMetricDescriptor {
name: INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
labels: SERVER_OPERATION_BACKEND_LABELS,
labels: OPERATION_BACKEND_LABELS,
},
];
fn current_server_label() -> &'static str {
static STABLE_SERVER_LABEL: OnceLock<String> = OnceLock::new();
static FALLBACK_SERVER_LABEL: LazyLock<String> = LazyLock::new(rustfs_utils::get_local_ip_with_default);
if let Some(server) = STABLE_SERVER_LABEL.get() {
return server.as_str();
}
if let Some(server) = rustfs_common::try_get_global_local_node_name() {
let _ = STABLE_SERVER_LABEL.set(server);
if let Some(server) = STABLE_SERVER_LABEL.get() {
return server.as_str();
}
}
FALLBACK_SERVER_LABEL.as_str()
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct InternodeMetricsSnapshot {
pub sent_bytes_total: u64,
@@ -213,7 +193,7 @@ impl InternodeMetrics {
return;
}
self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
counter!("rustfs_system_network_internode_sent_bytes_total", SERVER_LABEL => current_server_label()).increment(bytes);
counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
}
pub fn record_sent_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
@@ -227,13 +207,7 @@ impl InternodeMetrics {
if bytes == 0 {
return;
}
counter!(
INTERNODE_OPERATION_SENT_BYTES_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(bytes);
counter!(INTERNODE_OPERATION_SENT_BYTES_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(bytes);
}
pub fn record_recv_bytes(&self, bytes: usize) {
@@ -242,7 +216,7 @@ impl InternodeMetrics {
return;
}
self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
counter!("rustfs_system_network_internode_recv_bytes_total", SERVER_LABEL => current_server_label()).increment(bytes);
counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
}
pub fn record_recv_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
@@ -256,18 +230,12 @@ impl InternodeMetrics {
if bytes == 0 {
return;
}
counter!(
INTERNODE_OPERATION_RECV_BYTES_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(bytes);
counter!(INTERNODE_OPERATION_RECV_BYTES_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(bytes);
}
pub fn record_outgoing_request(&self) {
self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
counter!("rustfs_system_network_internode_requests_outgoing_total", SERVER_LABEL => current_server_label()).increment(1);
counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
}
pub fn record_outgoing_request_for_operation(&self, operation: &'static str) {
@@ -276,18 +244,13 @@ impl InternodeMetrics {
pub fn record_outgoing_request_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
self.record_outgoing_request();
counter!(
INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.increment(1);
}
pub fn record_incoming_request(&self) {
self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
counter!("rustfs_system_network_internode_requests_incoming_total", SERVER_LABEL => current_server_label()).increment(1);
counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
}
pub fn record_incoming_request_for_operation(&self, operation: &'static str) {
@@ -296,18 +259,13 @@ impl InternodeMetrics {
pub fn record_incoming_request_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
self.record_incoming_request();
counter!(
INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.increment(1);
}
pub fn record_error(&self) {
self.errors_total.fetch_add(1, Ordering::Relaxed);
counter!("rustfs_system_network_internode_errors_total", SERVER_LABEL => current_server_label()).increment(1);
counter!("rustfs_system_network_internode_errors_total").increment(1);
}
pub fn record_error_for_operation(&self, operation: &'static str) {
@@ -316,24 +274,13 @@ impl InternodeMetrics {
pub fn record_error_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
self.record_error();
counter!(
INTERNODE_OPERATION_ERRORS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
}
pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
let duration_ms = duration.as_secs_f64() * 1000.0;
metrics::histogram!(
INTERNODE_OPERATION_DURATION_MS,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.record(duration_ms);
metrics::histogram!(INTERNODE_OPERATION_DURATION_MS, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.record(duration_ms);
}
pub fn record_classified_error_for_operation_and_backend(
@@ -344,7 +291,6 @@ impl InternodeMetrics {
) {
counter!(
INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
CLASSIFICATION_LABEL => classification
@@ -360,7 +306,6 @@ impl InternodeMetrics {
) {
counter!(
INTERNODE_OPERATION_RETRIES_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
CLASSIFICATION_LABEL => classification
@@ -376,7 +321,6 @@ impl InternodeMetrics {
) {
counter!(
INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
CLASSIFICATION_LABEL => classification
@@ -393,7 +337,6 @@ impl InternodeMetrics {
self.operation_http_versions_total.fetch_add(1, Ordering::Relaxed);
counter!(
INTERNODE_OPERATION_HTTP_VERSIONS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend,
HTTP_VERSION_LABEL => http_version
@@ -403,24 +346,13 @@ impl InternodeMetrics {
pub fn record_stall_timeout_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
self.operation_stall_timeouts_total.fetch_add(1, Ordering::Relaxed);
counter!(
INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
}
pub fn record_write_shutdown_error_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
self.operation_write_shutdown_errors_total.fetch_add(1, Ordering::Relaxed);
counter!(
INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.increment(1);
}
/// Record the payload size (bytes) of a completed internode operation into a histogram
@@ -428,26 +360,15 @@ impl InternodeMetrics {
/// (`ReadAll`/`ReadMultiple`/`WriteAll`) would benefit from being moved off the shared
/// control-plane channel (see docs/grpc-optimization P1).
pub fn record_operation_payload_bytes(&self, operation: &'static str, backend: &'static str, bytes: usize) {
metrics::histogram!(
INTERNODE_OPERATION_PAYLOAD_BYTES,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.record(bytes as f64);
metrics::histogram!(INTERNODE_OPERATION_PAYLOAD_BYTES, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
.record(bytes as f64);
}
/// Increment the large-payload counter for an operation+backend whose payload exceeded the
/// caller-configured warning threshold. Feeds alerting on large unary RPCs that contend with
/// latency-sensitive control-plane traffic on the shared connection.
pub fn record_large_operation_payload(&self, operation: &'static str, backend: &'static str) {
counter!(
INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
SERVER_LABEL => current_server_label(),
OPERATION_LABEL => operation,
BACKEND_LABEL => backend
)
.increment(1);
counter!(INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
}
/// Count a decode that fell back to the JSON compatibility field because the msgpack `_bin`
@@ -456,20 +377,13 @@ impl InternodeMetrics {
/// dropped (grpc-optimization P2). `direction` is [`INTERNODE_MSGPACK_DIRECTION_REQUEST`] or
/// [`INTERNODE_MSGPACK_DIRECTION_RESPONSE`]; `message` is the low-cardinality value name.
pub fn record_msgpack_json_fallback(&self, direction: &'static str, message: &'static str) {
counter!(
INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL,
SERVER_LABEL => current_server_label(),
DIRECTION_LABEL => direction,
MESSAGE_LABEL => message
)
.increment(1);
counter!(INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL, DIRECTION_LABEL => direction, MESSAGE_LABEL => message).increment(1);
}
pub fn record_msgpack_json_decode(&self, direction: &'static str, message: &'static str, codec: &'static str) {
self.msgpack_json_decode_total.fetch_add(1, Ordering::Relaxed);
counter!(
INTERNODE_MSGPACK_JSON_DECODE_TOTAL,
SERVER_LABEL => current_server_label(),
DIRECTION_LABEL => direction,
MESSAGE_LABEL => message,
CODEC_LABEL => codec
@@ -481,7 +395,6 @@ impl InternodeMetrics {
self.msgpack_json_decode_error_total.fetch_add(1, Ordering::Relaxed);
counter!(
INTERNODE_MSGPACK_JSON_DECODE_ERROR_TOTAL,
SERVER_LABEL => current_server_label(),
DIRECTION_LABEL => direction,
MESSAGE_LABEL => message,
CODEC_LABEL => codec
@@ -507,7 +420,7 @@ impl InternodeMetrics {
/// enabled; after the strict flip the legacy fallback path is closed and the counter stays flat.
pub fn record_signature_v1_fallback(&self) {
self.signature_v1_fallback_total.fetch_add(1, Ordering::Relaxed);
counter!(INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
counter!(INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL).increment(1);
}
/// Count a mutating internode disk RPC that was accepted without a signature-bound canonical
@@ -518,14 +431,14 @@ impl InternodeMetrics {
/// mutations are rejected and the counter stays flat.
pub fn record_body_digest_fallback(&self) {
self.body_digest_fallback_total.fetch_add(1, Ordering::Relaxed);
counter!(INTERNODE_BODY_DIGEST_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
counter!(INTERNODE_BODY_DIGEST_FALLBACK_TOTAL).increment(1);
}
/// Count an accepted v1/v2 request that does not carry the replay-scoped signature. This is
/// the convergence signal for `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT`.
pub fn record_replay_scope_fallback(&self) {
self.replay_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
counter!(INTERNODE_REPLAY_SCOPE_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
counter!(INTERNODE_REPLAY_SCOPE_FALLBACK_TOTAL).increment(1);
}
/// Count a body-bound internode RPC rejected because the replay-protection nonce cache was
@@ -534,13 +447,12 @@ impl InternodeMetrics {
/// mutation rate and writes are being refused — alert on this counter.
pub fn record_replay_cache_overflow(&self) {
self.replay_cache_overflow_total.fetch_add(1, Ordering::Relaxed);
counter!(INTERNODE_REPLAY_CACHE_OVERFLOW_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
counter!(INTERNODE_REPLAY_CACHE_OVERFLOW_TOTAL).increment(1);
}
pub fn record_erasure_write_quorum_failure(&self, stage: &'static str, dominant_error: &'static str) {
counter!(
ERASURE_WRITE_QUORUM_FAILURES_TOTAL,
SERVER_LABEL => current_server_label(),
STAGE_LABEL => stage,
DOMINANT_ERROR_LABEL => dominant_error
)
@@ -552,12 +464,11 @@ impl InternodeMetrics {
self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
gauge!("rustfs_system_network_internode_dial_avg_time_nanos", SERVER_LABEL => current_server_label())
.set(total as f64 / samples as f64);
gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
if !success {
self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
counter!("rustfs_system_network_internode_dial_errors_total", SERVER_LABEL => current_server_label()).increment(1);
counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
}
let now_ms = SystemTime::now()
@@ -776,9 +687,6 @@ fn cluster_peer_health_keys() -> Vec<String> {
#[cfg(test)]
mod tests {
use super::*;
use metrics::with_local_recorder;
use metrics_util::debugging::DebuggingRecorder;
use std::collections::HashSet;
#[test]
fn snapshot_reports_recorded_values() {
@@ -842,22 +750,22 @@ mod tests {
fn operation_metric_descriptors_include_backend_and_operation_labels() {
assert_eq!(INTERNODE_OPERATION_METRICS.len(), 15);
for metric in &INTERNODE_OPERATION_METRICS[..6] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
}
for metric in &INTERNODE_OPERATION_METRICS[6..9] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
}
assert_eq!(
INTERNODE_OPERATION_METRICS[9].labels,
&[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL]
&[OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL]
);
for metric in &INTERNODE_OPERATION_METRICS[10..12] {
assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
}
assert_eq!(INTERNODE_OPERATION_METRICS[12].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[12].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
// Payload histogram + large-payload counter carry operation+backend labels.
assert_eq!(INTERNODE_OPERATION_METRICS[13].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[14].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[13].labels, &[OPERATION_LABEL, BACKEND_LABEL]);
assert_eq!(INTERNODE_OPERATION_METRICS[14].labels, &[OPERATION_LABEL, BACKEND_LABEL]);
}
#[test]
@@ -935,59 +843,6 @@ mod tests {
);
}
#[test]
fn direct_internode_metrics_emit_stable_server_label() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let metrics = InternodeMetrics::default();
with_local_recorder(&recorder, || {
metrics.record_sent_bytes_for_operation_and_backend(
INTERNODE_OPERATION_READ_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
128,
);
metrics.record_recv_bytes_for_operation_and_backend(
INTERNODE_OPERATION_PUT_FILE_STREAM,
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
256,
);
metrics.record_dial_result(Duration::from_millis(3), false);
});
let observed: Vec<(String, HashSet<String>, Option<String>)> = snapshotter
.snapshot()
.into_vec()
.into_iter()
.filter(|(composite, _, _, _)| {
matches!(
composite.key().name(),
"rustfs_system_network_internode_sent_bytes_total"
| "rustfs_system_network_internode_recv_bytes_total"
| INTERNODE_OPERATION_SENT_BYTES_TOTAL
| INTERNODE_OPERATION_RECV_BYTES_TOTAL
| "rustfs_system_network_internode_dial_avg_time_nanos"
| "rustfs_system_network_internode_dial_errors_total"
)
})
.map(|(composite, _, _, _)| {
let labels = composite.key().labels();
let keys = labels.clone().map(|label| label.key().to_string()).collect();
let server = labels
.filter(|label| label.key() == SERVER_LABEL)
.map(|label| label.value().to_string())
.next();
(composite.key().name().to_string(), keys, server)
})
.collect();
assert_eq!(observed.len(), 6);
for (name, keys, server) in observed {
assert!(keys.contains(SERVER_LABEL), "{name} must carry the server label");
assert!(server.is_some_and(|value| !value.is_empty()), "{name} server label must not be empty");
}
}
#[test]
fn msgpack_json_fallback_counter_records_without_panicking() {
// Smoke test: the counter accepts both directions and a static message label.
-27
View File
@@ -28,34 +28,7 @@ authors.workspace = true
[lints]
workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-credentials/hotpath",
"rustfs-policy/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
tokio = { workspace = true, features = ["rt", "sync"] }
reqwest = { workspace = true, features = ["json"] }
serde = { workspace = true, features = ["derive"] }
-2
View File
@@ -95,7 +95,6 @@ impl KeystoneClient {
}
/// Validate a Keystone token
#[hotpath::measure]
pub async fn validate_token(&self, token: &str) -> Result<KeystoneToken> {
match self.version {
KeystoneVersion::V3 => self.validate_token_v3(token).await,
@@ -239,7 +238,6 @@ impl KeystoneClient {
}
/// Get EC2 credentials for a user
#[hotpath::measure]
pub async fn get_ec2_credentials(&self, user_id: &str, project_id: Option<&str>) -> Result<Vec<EC2Credential>> {
let admin_token = self.get_admin_token().await?;
-22
View File
@@ -25,25 +25,3 @@ For local KMS end-to-end tests, keep proxy bypass settings:
NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= \
cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1
```
## Local Key Export for SSE-S3 Migration Tests
Use the read-only `local_kms_key_decrypt` example to export an AES-256 Local
KMS key as the base64 value expected by `RUSTFS_SSE_S3_MASTER_KEY`:
```bash
export RUSTFS_KMS_LOCAL_MASTER_KEY='<local-kms-at-rest-master-key>'
export RUSTFS_SSE_S3_MASTER_KEY="$(
cargo run -q -p rustfs-kms --example local_kms_key_decrypt -- \
/absolute/path/to/<key-id>.key
)"
```
For a `plaintext-dev-only` Local KMS key file,
`RUSTFS_KMS_LOCAL_MASTER_KEY` is not required.
The example writes only the base64-encoded 32-byte key to stdout. Diagnostics
go to stderr. Never paste its output into logs, shell history, issue comments,
or committed configuration. The export path must remain read-only and must
reuse `LocalKmsClient` decoding so current Argon2id and legacy key-file
compatibility stay aligned with the backend.
-26
View File
@@ -28,7 +28,6 @@ categories = ["cryptography", "web-programming", "authentication"]
workspace = true
[dependencies]
hotpath.workspace = true
# Core dependencies
async-trait = { workspace = true }
tokio = { workspace = true, features = ["fs", "io-util", "macros", "rt-multi-thread", "sync", "time"] }
@@ -38,8 +37,6 @@ serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["raw_value"] }
tracing = { workspace = true }
thiserror = { workspace = true }
# Operation metrics emitted by the retry policy engine (crate::policy).
metrics = { workspace = true }
# Cryptography
aes-gcm = { workspace = true, features = ["rand_core"] }
@@ -68,34 +65,11 @@ rustfs-security-governance = { workspace = true }
# HTTP client for Vault
reqwest = { workspace = true }
vaultrs = { workspace = true }
# vaultrs surfaces transport-level failures as wrapped rustify errors; the
# operation policy needs the concrete type to classify them for retry decisions.
rustify = { workspace = true }
tokio-util = { workspace = true }
[dev-dependencies]
anyhow = { workspace = true }
# Debugging recorder for asserting emitted metrics in tests.
metrics-util = { version = "0.20", features = ["debugging"] }
insta = { workspace = true, features = ["yaml", "json"] }
tempfile = { workspace = true }
temp-env = { workspace = true }
# "net" backs the scripted loopback Vault used by the policy wiring tests.
tokio = { workspace = true, features = ["net", "test-util"] }
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/reqwest-0-13",
"rustfs-security-governance/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-security-governance/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-security-governance/hotpath-cpu", "rustfs-utils/hotpath-cpu"]
@@ -1,112 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
use rustfs_kms::{LocalConfig, backends::local::LocalKmsClient};
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use zeroize::Zeroizing;
const LOCAL_KMS_MASTER_KEY_ENV: &str = "RUSTFS_KMS_LOCAL_MASTER_KEY";
fn usage(program: &str) -> String {
format!(
"Usage: {program} <local-kms-key-file>\n\
Reads {LOCAL_KMS_MASTER_KEY_ENV} when the key file is encrypted.\n\
Writes only the base64-encoded 32-byte key to stdout."
)
}
fn resolve_key_file(path: &Path) -> Result<(PathBuf, String), String> {
let canonical = std::fs::canonicalize(path).map_err(|error| format!("cannot open Local KMS key file: {error}"))?;
if canonical.extension().and_then(|extension| extension.to_str()) != Some("key") {
return Err("Local KMS key file must have a .key extension".to_string());
}
let key_dir = canonical
.parent()
.ok_or_else(|| "Local KMS key file must have a parent directory".to_string())?
.to_path_buf();
let key_id = canonical
.file_stem()
.and_then(|stem| stem.to_str())
.filter(|stem| !stem.is_empty())
.ok_or_else(|| "Local KMS key file name must contain a valid UTF-8 key ID".to_string())?
.to_string();
Ok((key_dir, key_id))
}
async fn run() -> Result<(), String> {
let mut args = std::env::args();
let program = args.next().unwrap_or_else(|| "local_kms_key_decrypt".to_string());
let Some(key_file) = args.next() else {
return Err(usage(&program));
};
if args.next().is_some() {
return Err(usage(&program));
}
let (key_dir, key_id) = resolve_key_file(Path::new(&key_file))?;
let master_key = std::env::var(LOCAL_KMS_MASTER_KEY_ENV).ok().filter(|value| !value.is_empty());
let client = LocalKmsClient::new_for_key_export(LocalConfig {
key_dir,
master_key,
file_permissions: Some(0o600),
})
.await
.map_err(|error| error.to_string())?;
let key_material = client
.decrypt_key_material_for_export(&key_id)
.await
.map_err(|error| error.to_string())?;
let encoded = Zeroizing::new(BASE64_STANDARD.encode(key_material.as_ref()));
let mut stdout = io::stdout().lock();
writeln!(stdout, "{}", encoded.as_str()).map_err(|error| format!("failed to write decrypted key: {error}"))
}
#[tokio::main]
async fn main() {
if let Err(error) = run().await {
let _ = writeln!(io::stderr().lock(), "local_kms_key_decrypt: {error}");
std::process::exit(1);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolve_key_file_extracts_directory_and_key_id() {
let directory = tempfile::tempdir().expect("create temporary directory");
let key_file = directory.path().join("migration-key.key");
std::fs::write(&key_file, b"{}").expect("create key file");
let (key_dir, key_id) = resolve_key_file(&key_file).expect("resolve key file");
assert_eq!(key_dir, directory.path().canonicalize().expect("canonical directory"));
assert_eq!(key_id, "migration-key");
}
#[test]
fn resolve_key_file_rejects_non_key_extension() {
let directory = tempfile::tempdir().expect("create temporary directory");
let key_file = directory.path().join("migration-key.json");
std::fs::write(&key_file, b"{}").expect("create key file");
let error = resolve_key_file(&key_file).expect_err("non-key file must be rejected");
assert!(error.contains(".key"));
}
}
+12 -102
View File
@@ -71,10 +71,7 @@ impl fmt::Debug for ConfigureLocalKmsRequest {
}
}
/// Request to configure KMS with the Vault KV v2 storage backend.
///
/// This backend stores master key material directly in KV v2; confidentiality relies on
/// Vault ACLs and KV v2 at-rest encryption, with no Transit wrapping involved.
/// Request to configure KMS with Vault KV v2 + Transit backend
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ConfigureVaultKmsRequest {
@@ -85,8 +82,7 @@ pub struct ConfigureVaultKmsRequest {
pub auth_method: VaultAuthMethod,
/// Vault namespace (Vault Enterprise, optional)
pub namespace: Option<String>,
/// Deprecated: legacy Transit engine mount path. Still accepted so older clients keep
/// working, but the Vault KV2 backend never uses it.
/// Transit engine mount path
pub mount_path: Option<String>,
/// KV engine mount path for storing keys
pub kv_mount: Option<String>,
@@ -196,7 +192,7 @@ pub enum ConfigureKmsRequest {
/// Configure with Local backend
#[serde(alias = "local", alias = "Local")]
Local(ConfigureLocalKmsRequest),
/// Configure with the Vault KV v2 storage backend
/// Configure with Vault KV v2 + Transit backend
#[serde(
rename = "VaultKV2",
alias = "Vault",
@@ -235,55 +231,15 @@ pub struct StartKmsRequest {
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
enum StrictVaultAuthMethod {
Token {
token: String,
},
AppRole {
role_id: String,
#[serde(default)]
secret_id: String,
#[serde(default)]
secret_id_file: Option<std::path::PathBuf>,
#[serde(default)]
mount: Option<String>,
#[serde(default)]
refresh_safety_window_secs: Option<u64>,
},
TokenFile {
path: std::path::PathBuf,
#[serde(default)]
poll_interval_secs: Option<u64>,
#[serde(default)]
refresh_safety_window_secs: Option<u64>,
},
Token { token: String },
AppRole { role_id: String, secret_id: String },
}
impl From<StrictVaultAuthMethod> for VaultAuthMethod {
fn from(value: StrictVaultAuthMethod) -> Self {
match value {
StrictVaultAuthMethod::Token { token } => Self::Token { token },
StrictVaultAuthMethod::AppRole {
role_id,
secret_id,
secret_id_file,
mount,
refresh_safety_window_secs,
} => Self::AppRole {
role_id,
secret_id,
secret_id_file,
mount: mount.unwrap_or_else(|| crate::config::DEFAULT_VAULT_APPROLE_MOUNT.to_string()),
refresh_safety_window_secs,
},
StrictVaultAuthMethod::TokenFile {
path,
poll_interval_secs,
refresh_safety_window_secs,
} => Self::TokenFile {
path,
poll_interval_secs,
refresh_safety_window_secs,
},
StrictVaultAuthMethod::AppRole { role_id, secret_id } => Self::AppRole { role_id, secret_id },
}
}
}
@@ -377,7 +333,7 @@ pub enum BackendSummary {
/// File permissions (octal)
file_permissions: Option<u32>,
},
/// Vault KV v2 storage backend summary
/// Vault KV v2 + Transit backend summary
#[serde(alias = "vault")]
VaultKv2 {
/// Vault server address
@@ -388,8 +344,7 @@ pub enum BackendSummary {
has_stored_credentials: bool,
/// Namespace (if configured)
namespace: Option<String>,
/// Deprecated: legacy Transit mount path. Unused by the backend; kept only so the
/// serialized response shape stays stable for existing consumers.
/// Transit engine mount path
mount_path: String,
/// KV engine mount path
kv_mount: String,
@@ -443,7 +398,6 @@ impl From<&KmsConfig> for KmsConfigSummary {
auth_method_type: match &vault_config.auth_method {
VaultAuthMethod::Token { .. } => "token".to_string(),
VaultAuthMethod::AppRole { .. } => "approle".to_string(),
VaultAuthMethod::TokenFile { .. } => "token_file".to_string(),
},
has_stored_credentials: true,
namespace: vault_config.namespace.clone(),
@@ -457,7 +411,6 @@ impl From<&KmsConfig> for KmsConfigSummary {
auth_method_type: match &vault_config.auth_method {
VaultAuthMethod::Token { .. } => "token".to_string(),
VaultAuthMethod::AppRole { .. } => "approle".to_string(),
VaultAuthMethod::TokenFile { .. } => "token_file".to_string(),
},
has_stored_credentials: true,
namespace: vault_config.namespace.clone(),
@@ -668,52 +621,6 @@ mod tests {
}
}
#[test]
fn test_deserialize_vault_kv2_configure_request_mount_path_optional_but_accepted() {
// deny_unknown_fields regression guard: mount_path is deprecated but must remain
// accepted so older clients that still send it do not get a 400.
let with_mount_path = serde_json::json!({
"backend_type": "VaultKV2",
"address": "http://127.0.0.1:8200",
"auth_method": { "Token": { "token": "dev-root-token" } },
"mount_path": "transit"
});
let request: ConfigureKmsRequest =
serde_json::from_value(with_mount_path).expect("request with deprecated mount_path should deserialize");
let config = request.to_kms_config();
assert_eq!(config.vault_config().expect("vault-kv2 config").mount_path, "transit");
let without_mount_path = serde_json::json!({
"backend_type": "VaultKV2",
"address": "http://127.0.0.1:8200",
"auth_method": { "Token": { "token": "dev-root-token" } }
});
let request: ConfigureKmsRequest =
serde_json::from_value(without_mount_path).expect("request without mount_path should deserialize");
let config = request.to_kms_config();
assert_eq!(config.vault_config().expect("vault-kv2 config").mount_path, "transit");
}
#[test]
fn test_vault_kv2_status_summary_does_not_mention_transit() {
let config = KmsConfig::vault(
url::Url::parse("https://vault.example.com:8200").expect("vault URL"),
"summary-token".to_string(),
);
let response = KmsStatusResponse {
status: KmsServiceStatus::Running,
backend_type: Some(config.backend.clone()),
healthy: Some(true),
config_summary: Some(KmsConfigSummary::from(&config)),
};
let json = serde_json::to_string(&response).expect("kms status response should serialize");
assert!(
!json.contains("Transit"),
"vault-kv2 status output must not describe the backend as Transit: {json}"
);
}
#[test]
fn test_deserialize_vault_transit_configure_request() {
let cases = ["VaultTransit", "vault-transit", "vault_transit"];
@@ -914,7 +821,10 @@ mod tests {
});
let approle = ConfigureKmsRequest::VaultKv2(ConfigureVaultKmsRequest {
address: "https://vault.example.com:8200".to_string(),
auth_method: VaultAuthMethod::approle("configure-role-id".to_string(), "configure-approle-secret-id".to_string()),
auth_method: VaultAuthMethod::AppRole {
role_id: "configure-role-id".to_string(),
secret_id: "configure-approle-secret-id".to_string(),
},
namespace: None,
mount_path: Some("transit".to_string()),
kv_mount: Some("secret".to_string()),
-345
View File
@@ -1,345 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Shared key state × operation contract tests for KMS backends.
//!
//! Every stateful backend must satisfy the same lifecycle matrix (see
//! `ensure_key_state_permits`): Enabled permits everything, Disabled permits
//! decryption and lifecycle recovery but rejects new cryptographic use, and
//! PendingDeletion rejects everything except decryption and cancellation.
//! Decryption staying available in Disabled/PendingDeletion is an explicit,
//! tested deviation from AWS KMS: disabling a key must not break reads of
//! objects already encrypted under it.
//!
//! The full matrix runs offline against the Local backend. The Vault KV2 and
//! Vault Transit runs exercise the same helper but need a live Vault dev
//! server, so they are `#[ignore]`d in CI. Static is covered by its own
//! stateless contract below.
use super::KmsBackend;
use super::local::LocalKmsBackend;
use super::static_kms::StaticKmsBackend;
use super::vault::VaultKmsBackend;
use super::vault_transit::VaultTransitKmsBackend;
use crate::config::KmsConfig;
use crate::error::{KmsError, Result};
use crate::manager::KmsManager;
use crate::service::ObjectEncryptionService;
use crate::types::{
CancelKeyDeletionRequest, CreateKeyRequest, DecryptRequest, DeleteKeyRequest, DescribeKeyRequest, EncryptRequest,
GenerateDataKeyRequest, KeySpec, KeyState, KeyUsage, ObjectEncryptionContext,
};
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use rand::RngExt as _;
use std::collections::HashMap;
use std::sync::Arc;
fn expect_unsupported<T: std::fmt::Debug>(result: Result<T>) {
match result {
Err(KmsError::UnsupportedCapability { .. }) => {}
other => panic!("expected UnsupportedCapability, got {other:?}"),
}
}
/// Rotation while not Enabled: backends with rotation support must reject it
/// through the state machine; backends without it report the capability gap.
async fn expect_rotate_rejected(backend: &dyn KmsBackend, key_id: &str) {
let result = backend.rotate_key(key_id).await;
if backend.capabilities().rotate {
expect_invalid_key_state(result, "");
} else {
expect_unsupported(result);
}
}
fn expect_invalid_key_state<T: std::fmt::Debug>(result: Result<T>, expected_fragment: &str) {
match result {
Err(KmsError::InvalidOperation { message }) => assert!(
message.contains(expected_fragment),
"expected invalid-key-state message containing {expected_fragment:?}, got {message:?}"
),
other => panic!("expected InvalidOperation (invalid key state), got {other:?}"),
}
}
fn context() -> HashMap<String, String> {
HashMap::from([("bucket".to_string(), "contract".to_string())])
}
fn generate_request(key_id: &str) -> GenerateDataKeyRequest {
GenerateDataKeyRequest {
key_id: key_id.to_string(),
key_spec: KeySpec::Aes256,
encryption_context: context(),
}
}
fn encrypt_request(key_id: &str) -> EncryptRequest {
EncryptRequest {
key_id: key_id.to_string(),
plaintext: b"contract-plaintext".to_vec(),
encryption_context: context(),
grant_tokens: Vec::new(),
}
}
fn decrypt_request(ciphertext: Vec<u8>) -> DecryptRequest {
DecryptRequest {
ciphertext,
encryption_context: context(),
grant_tokens: Vec::new(),
}
}
fn schedule_request(key_id: &str) -> DeleteKeyRequest {
DeleteKeyRequest {
key_id: key_id.to_string(),
pending_window_in_days: Some(7),
force_immediate: None,
}
}
fn cancel_request(key_id: &str) -> CancelKeyDeletionRequest {
CancelKeyDeletionRequest {
key_id: key_id.to_string(),
}
}
fn create_request(key_name: String) -> CreateKeyRequest {
CreateKeyRequest {
key_name: Some(key_name),
key_usage: KeyUsage::EncryptDecrypt,
..Default::default()
}
}
async fn assert_key_state(backend: &dyn KmsBackend, key_id: &str, expected: KeyState) {
let described = backend
.describe_key(DescribeKeyRequest {
key_id: key_id.to_string(),
})
.await
.expect("describe_key must succeed for an existing key");
assert_eq!(described.key_metadata.key_state, expected, "unexpected state for key {key_id}");
}
/// Drives one freshly created (Enabled) key through the full state matrix,
/// entirely through the `KmsBackend` product surface.
async fn assert_state_machine_contract(backend: &dyn KmsBackend, key_id: &str) {
// Enabled: cryptographic use is allowed. Keep an envelope around to prove
// decryption keeps working in later states.
let data_key = backend
.generate_data_key(generate_request(key_id))
.await
.expect("Enabled key must generate data keys");
backend
.encrypt(encrypt_request(key_id))
.await
.expect("Enabled key must encrypt");
// Enabled -> Disabled.
backend.disable_key(key_id).await.expect("disable from Enabled must succeed");
assert_key_state(backend, key_id, KeyState::Disabled).await;
// Disabled: new cryptographic use and rotation are rejected...
expect_invalid_key_state(backend.encrypt(encrypt_request(key_id)).await, "disabled");
expect_invalid_key_state(backend.generate_data_key(generate_request(key_id)).await, "disabled");
expect_rotate_rejected(backend, key_id).await;
// ...but decryption of existing data keeps working (explicit AWS deviation)...
let decrypted = backend
.decrypt(decrypt_request(data_key.ciphertext_blob.clone()))
.await
.expect("decrypt with a disabled key must keep working");
assert_eq!(decrypted.plaintext, data_key.plaintext_key, "decrypt must recover the original data key");
// ...disable stays idempotent, cancel has nothing to cancel, and enable recovers.
backend.disable_key(key_id).await.expect("disable must be idempotent");
expect_invalid_key_state(backend.cancel_key_deletion(cancel_request(key_id)).await, "not pending deletion");
backend.enable_key(key_id).await.expect("enable from Disabled must succeed");
assert_key_state(backend, key_id, KeyState::Enabled).await;
// Disabled keys may still be scheduled for deletion.
backend
.disable_key(key_id)
.await
.expect("disable before scheduling must succeed");
backend
.delete_key(schedule_request(key_id))
.await
.expect("scheduling deletion of a disabled key must succeed");
assert_key_state(backend, key_id, KeyState::PendingDeletion).await;
// PendingDeletion: everything except decryption and cancellation is rejected.
expect_invalid_key_state(backend.encrypt(encrypt_request(key_id)).await, "pending deletion");
expect_invalid_key_state(backend.generate_data_key(generate_request(key_id)).await, "pending deletion");
expect_invalid_key_state(backend.enable_key(key_id).await, "pending deletion");
expect_invalid_key_state(backend.disable_key(key_id).await, "pending deletion");
expect_rotate_rejected(backend, key_id).await;
expect_invalid_key_state(backend.delete_key(schedule_request(key_id)).await, "pending deletion");
let decrypted = backend
.decrypt(decrypt_request(data_key.ciphertext_blob.clone()))
.await
.expect("decrypt with a pending-deletion key must keep working");
assert_eq!(decrypted.plaintext, data_key.plaintext_key);
// PendingDeletion -> Enabled through cancellation.
backend
.cancel_key_deletion(cancel_request(key_id))
.await
.expect("cancel from PendingDeletion must succeed");
assert_key_state(backend, key_id, KeyState::Enabled).await;
backend
.generate_data_key(generate_request(key_id))
.await
.expect("cancelled key must be usable again");
// Cancel without a pending deletion is an invalid state transition.
expect_invalid_key_state(backend.cancel_key_deletion(cancel_request(key_id)).await, "not pending deletion");
}
async fn local_fixture() -> (tempfile::TempDir, KmsConfig, LocalKmsBackend, String) {
let temp_dir = tempfile::tempdir().expect("temp dir should be created");
let config = KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults();
let backend = LocalKmsBackend::new(config.clone())
.await
.expect("local backend should build");
let created = backend
.create_key(create_request("contract-key".to_string()))
.await
.expect("key should be created");
(temp_dir, config, backend, created.key_id)
}
#[tokio::test]
async fn local_backend_state_machine_contract() {
let (_temp_dir, _config, backend, key_id) = local_fixture().await;
assert_state_machine_contract(&backend, &key_id).await;
}
/// SSE-shaped regression: disabling a key must not break decryption of data
/// keys created while it was enabled, while new data key creation must fail.
#[tokio::test]
async fn local_disabled_key_keeps_decrypting_existing_envelopes() {
let (_temp_dir, config, backend, key_id) = local_fixture().await;
let backend = Arc::new(backend);
let service = ObjectEncryptionService::new(KmsManager::new(backend.clone(), config));
let object_context = ObjectEncryptionContext::new("sse-bucket".to_string(), "dir/object.bin".to_string());
let kms_key = Some(key_id.clone());
let (_data_key, encrypted_blob) = service
.create_data_key(&kms_key, &object_context)
.await
.expect("data key creation must succeed while the key is enabled");
backend
.lifecycle_client()
.disable_key(&key_id, None)
.await
.expect("disable must succeed");
service
.decrypt_data_key(&encrypted_blob, &object_context)
.await
.expect("existing objects must stay readable after their KMS key is disabled");
expect_invalid_key_state(service.create_data_key(&kms_key, &object_context).await, "disabled");
}
/// Static is a stateless read-only backend: cryptographic operations always
/// work against the single configured key and every lifecycle mutation is
/// rejected as an invalid operation.
#[tokio::test]
async fn static_backend_stateless_contract() {
let key_id = "static-contract-key";
let mut raw_key = [0u8; 32];
rand::rng().fill(&mut raw_key[..]);
let config = KmsConfig::static_kms(key_id.to_string(), BASE64.encode(raw_key));
let static_backend = StaticKmsBackend::new(config).await.expect("static backend should build");
let backend: &dyn KmsBackend = &static_backend;
let data_key = backend
.generate_data_key(generate_request(key_id))
.await
.expect("static backend must generate data keys");
let decrypted = backend
.decrypt(decrypt_request(data_key.ciphertext_blob.clone()))
.await
.expect("static backend must decrypt its own envelopes");
assert_eq!(decrypted.plaintext, data_key.plaintext_key);
assert_key_state(backend, key_id, KeyState::Enabled).await;
expect_invalid_key_state(backend.create_key(create_request("another-key".to_string())).await, "read-only");
expect_invalid_key_state(backend.delete_key(schedule_request(key_id)).await, "read-only");
expect_invalid_key_state(backend.cancel_key_deletion(cancel_request(key_id)).await, "read-only");
// Enable/disable and rotation are capability gaps at the product
// surface, not state-machine rejections.
expect_unsupported(backend.enable_key(key_id).await);
expect_unsupported(backend.disable_key(key_id).await);
expect_unsupported(backend.rotate_key(key_id).await);
}
fn vault_dev_config(constructor: fn(url::Url, String) -> KmsConfig) -> KmsConfig {
let address = std::env::var("RUSTFS_KMS_VAULT_ADDR").unwrap_or_else(|_| "http://127.0.0.1:8200".to_string());
let token = std::env::var("RUSTFS_KMS_VAULT_TOKEN").unwrap_or_else(|_| "dev-token".to_string());
let mut config = constructor(url::Url::parse(&address).expect("vault address should parse"), token);
config.allow_insecure_dev_defaults = true;
config
}
#[tokio::test]
#[ignore] // Requires a running Vault instance (dev mode) with a KV2 mount
async fn vault_kv2_backend_state_machine_contract() {
let config = vault_dev_config(KmsConfig::vault);
let backend = VaultKmsBackend::new(config).await.expect("vault kv2 backend should build");
let created = backend
.create_key(create_request(format!("contract-{}", uuid::Uuid::new_v4())))
.await
.expect("key should be created");
assert_state_machine_contract(&backend, &created.key_id).await;
// KV2 additionally supports version-retaining rotation, which must only
// work while the key is Enabled (the shared matrix covered the
// rejections).
backend
.rotate_key(&created.key_id)
.await
.expect("rotation of an Enabled KV2 key must succeed");
// Cleanup: leave the key pending deletion so repeated runs stay tidy.
let _ = backend.delete_key(schedule_request(&created.key_id)).await;
}
#[tokio::test]
#[ignore] // Requires a running Vault instance (dev mode) with the transit engine enabled
async fn vault_transit_backend_state_machine_contract() {
let config = vault_dev_config(KmsConfig::vault_transit);
let backend = VaultTransitKmsBackend::new(config)
.await
.expect("vault transit backend should build");
let created = backend
.create_key(create_request(format!("contract-{}", uuid::Uuid::new_v4())))
.await
.expect("key should be created");
assert_state_machine_contract(&backend, &created.key_id).await;
// Transit additionally supports rotation, which must only work while the
// key is Enabled (the shared matrix already covered the rejections).
backend
.rotate_key(&created.key_id)
.await
.expect("rotation of an Enabled transit key must succeed");
let _ = backend.delete_key(schedule_request(&created.key_id)).await;
}
File diff suppressed because it is too large Load Diff
+169 -377
View File
@@ -14,88 +14,144 @@
//! KMS backend implementations
use crate::error::{KmsError, Result};
use crate::error::Result;
use crate::types::*;
use async_trait::async_trait;
use jiff::Zoned;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
#[cfg(test)]
mod contract_tests;
pub mod local;
#[cfg(test)]
pub(crate) mod scripted_vault;
pub mod static_kms;
pub mod vault;
pub(crate) mod vault_credentials;
pub mod vault_transit;
/// Operations whose availability depends on the key's lifecycle state.
///
/// Decryption is deliberately absent: RustFS allows decryption with
/// `Disabled` and `PendingDeletion` keys — an explicit deviation from AWS
/// KMS — because rejecting it would break reads of every object encrypted
/// under a key the moment it is disabled. Deletion cancellation is also
/// absent: it is valid exactly when the key is `PendingDeletion`, which call
/// sites enforce directly.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum StateGatedOperation {
Encrypt,
GenerateDataKey,
Rotate,
Enable,
Disable,
ScheduleDeletion,
}
/// Abstract KMS client interface that all backends must implement
#[async_trait]
pub trait KmsClient: Send + Sync {
/// Generate a new data encryption key (DEK)
///
/// Creates a new data key using the specified master key. The returned DataKey
/// contains both the plaintext and encrypted versions of the key.
///
/// # Arguments
/// * `request` - The key generation request
/// * `context` - Optional operation context for auditing
///
/// # Returns
/// Returns a DataKey containing both plaintext and encrypted key material
async fn generate_data_key(&self, request: &GenerateKeyRequest, context: Option<&OperationContext>) -> Result<DataKeyInfo>;
impl StateGatedOperation {
fn describe(self) -> &'static str {
match self {
Self::Encrypt => "encryption",
Self::GenerateDataKey => "data key generation",
Self::Rotate => "rotation",
Self::Enable => "enabling",
Self::Disable => "disabling",
Self::ScheduleDeletion => "deletion scheduling",
}
}
}
/// Encrypt data directly using a master key
///
/// Encrypts the provided plaintext using the specified master key.
/// This is different from generate_data_key as it encrypts user data directly.
///
/// # Arguments
/// * `request` - The encryption request containing plaintext and key ID
/// * `context` - Optional operation context for auditing
async fn encrypt(&self, request: &EncryptRequest, context: Option<&OperationContext>) -> Result<EncryptResponse>;
/// Enforce the shared key state × operation matrix.
///
/// - `Enabled`: every operation is allowed.
/// - `Disabled`: enabling, disabling (idempotent) and deletion scheduling are
/// allowed; encryption, data key generation and rotation are rejected.
/// - `PendingDeletion`: every state-gated operation is rejected, including a
/// repeated deletion schedule; only cancellation and decryption proceed.
/// - `PendingImport`/`Unavailable`: the key is not usable and is reported as
/// not found.
pub(crate) fn ensure_key_state_permits(key_id: &str, state: &KeyState, operation: StateGatedOperation) -> Result<()> {
match state {
KeyState::Enabled => Ok(()),
KeyState::Disabled => match operation {
StateGatedOperation::Enable | StateGatedOperation::Disable | StateGatedOperation::ScheduleDeletion => Ok(()),
StateGatedOperation::Encrypt | StateGatedOperation::GenerateDataKey | StateGatedOperation::Rotate => Err(
KmsError::invalid_key_state(format!("Key {key_id} is disabled: {} is not allowed", operation.describe())),
),
},
KeyState::PendingDeletion => Err(KmsError::invalid_key_state(format!(
"Key {key_id} is pending deletion: {} is not allowed",
operation.describe()
))),
KeyState::PendingImport | KeyState::Unavailable => Err(KmsError::key_not_found(key_id)),
}
}
/// Decrypt data using a master key
///
/// Decrypts the provided ciphertext. The KMS automatically determines
/// which key was used for encryption based on the ciphertext metadata.
///
/// # Arguments
/// * `request` - The decryption request containing ciphertext
/// * `context` - Optional operation context for auditing
async fn decrypt(&self, request: &DecryptRequest, context: Option<&OperationContext>) -> Result<Vec<u8>>;
/// [`ensure_key_state_permits`] for backends that persist [`KeyStatus`].
pub(crate) fn ensure_key_status_permits(key_id: &str, status: &KeyStatus, operation: StateGatedOperation) -> Result<()> {
let state = match status {
KeyStatus::Active => KeyState::Enabled,
KeyStatus::Disabled => KeyState::Disabled,
KeyStatus::PendingDeletion => KeyState::PendingDeletion,
KeyStatus::Deleted => KeyState::Unavailable,
};
ensure_key_state_permits(key_id, &state, operation)
/// Create a new master key
///
/// Creates a new master key in the KMS with the specified ID.
/// Returns an error if a key with the same ID already exists.
///
/// # Arguments
/// * `key_id` - Unique identifier for the new key
/// * `algorithm` - Key algorithm (e.g., "AES_256")
/// * `context` - Optional operation context for auditing
async fn create_key(&self, key_id: &str, algorithm: &str, context: Option<&OperationContext>) -> Result<MasterKeyInfo>;
/// Get information about a specific key
///
/// Returns metadata and information about the specified key.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `context` - Optional operation context for auditing
async fn describe_key(&self, key_id: &str, context: Option<&OperationContext>) -> Result<KeyInfo>;
/// List available keys
///
/// Returns a paginated list of keys available in the KMS.
///
/// # Arguments
/// * `request` - List request parameters (pagination, filters)
/// * `context` - Optional operation context for auditing
async fn list_keys(&self, request: &ListKeysRequest, context: Option<&OperationContext>) -> Result<ListKeysResponse>;
/// Enable a key
///
/// Enables a previously disabled key, allowing it to be used for cryptographic operations.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `context` - Optional operation context for auditing
async fn enable_key(&self, key_id: &str, context: Option<&OperationContext>) -> Result<()>;
/// Disable a key
///
/// Disables a key, preventing it from being used for new cryptographic operations.
/// Existing encrypted data can still be decrypted.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `context` - Optional operation context for auditing
async fn disable_key(&self, key_id: &str, context: Option<&OperationContext>) -> Result<()>;
/// Schedule key deletion
///
/// Schedules a key for deletion after a specified number of days.
/// This allows for a grace period to recover the key if needed.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `pending_window_days` - Number of days before actual deletion
/// * `context` - Optional operation context for auditing
async fn schedule_key_deletion(
&self,
key_id: &str,
pending_window_days: u32,
context: Option<&OperationContext>,
) -> Result<()>;
/// Cancel key deletion
///
/// Cancels a previously scheduled key deletion.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `context` - Optional operation context for auditing
async fn cancel_key_deletion(&self, key_id: &str, context: Option<&OperationContext>) -> Result<()>;
/// Rotate a key
///
/// Creates a new version of the specified key. Previous versions remain
/// available for decryption but new operations will use the new version.
///
/// # Arguments
/// * `key_id` - The key identifier
/// * `context` - Optional operation context for auditing
async fn rotate_key(&self, key_id: &str, context: Option<&OperationContext>) -> Result<MasterKeyInfo>;
/// Health check
///
/// Performs a health check on the KMS backend to ensure it's operational.
async fn health_check(&self) -> Result<()>;
/// Get backend information
///
/// Returns information about the KMS backend (type, version, etc.).
fn backend_info(&self) -> BackendInfo;
}
/// Simplified KMS backend interface for manager
@@ -125,322 +181,58 @@ pub trait KmsBackend: Send + Sync {
/// Cancel key deletion
async fn cancel_key_deletion(&self, request: CancelKeyDeletionRequest) -> Result<CancelKeyDeletionResponse>;
/// Enable a disabled key so it can be used for cryptographic operations
/// again.
///
/// Backends that advertise [`BackendCapabilities::enable_disable`] must
/// override this method; the default rejects the operation.
async fn enable_key(&self, _key_id: &str) -> Result<()> {
Err(KmsError::unsupported_capability("backend without enable/disable support", "enable_key"))
}
/// Disable a key, rejecting new cryptographic use while existing data
/// remains decryptable.
///
/// Backends that advertise [`BackendCapabilities::enable_disable`] must
/// override this method; the default rejects the operation.
async fn disable_key(&self, _key_id: &str) -> Result<()> {
Err(KmsError::unsupported_capability("backend without enable/disable support", "disable_key"))
}
/// Rotate a key to a new version while prior versions remain available
/// for decryption.
///
/// Only backends that advertise [`BackendCapabilities::rotate`] (that is,
/// backends with retained version history) may override this method; the
/// default rejects the operation.
async fn rotate_key(&self, _key_id: &str) -> Result<()> {
Err(KmsError::unsupported_capability("backend without rotation support", "rotate_key"))
}
/// Health check
async fn health_check(&self) -> Result<bool>;
/// Report which operations this backend actually supports.
///
/// The default is conservative: only the operations every backend is
/// required to implement by this trait are advertised. Optional lifecycle
/// operations (rotation, enable/disable, deletion scheduling, ...) must be
/// opted in by overriding this method.
fn capabilities(&self) -> BackendCapabilities {
BackendCapabilities::minimal()
}
/// Remove a key whose scheduled deletion deadline has passed.
///
/// Used by the background deletion worker. Implementations must re-check
/// state and deadline under their own write synchronization so that a
/// concurrent cancellation observed after the caller's inspection wins
/// ([`ExpiredKeyRemoval::StateChanged`]), must write a tombstone (a
/// `Deleted`/`Unavailable` record) before destroying material so a crashed
/// removal can simply be re-run, and must treat an already-removed key as
/// success so the operation stays idempotent across restarts and nodes.
///
/// The default rejects the operation for backends without deletion
/// support.
async fn remove_expired_key(&self, _key_id: &str, _now: &Zoned) -> Result<ExpiredKeyRemoval> {
Err(KmsError::unsupported_capability("backend without deletion support", "remove_expired_key"))
}
}
/// Outcome of [`KmsBackend::remove_expired_key`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ExpiredKeyRemoval {
/// The key's record and material were removed, or were already gone.
Removed,
/// The key is no longer pending deletion (for example the deletion was
/// cancelled after the caller inspected it); nothing was removed.
StateChanged,
/// The key is pending deletion but its deadline has not passed, or it has
/// no persisted deadline (legacy record) and is never auto-removed.
NotExpired,
/// Information about a KMS backend
#[derive(Debug, Clone)]
pub struct BackendInfo {
/// Backend type name (e.g., "local", "vault")
pub backend_type: String,
/// Backend version
pub version: String,
/// Backend endpoint or location
pub endpoint: String,
/// Whether the backend is currently healthy
pub healthy: bool,
/// Additional metadata about the backend
pub metadata: HashMap<String, String>,
}
/// Set of operations a KMS backend supports.
///
/// Reported by [`KmsBackend::capabilities`] so callers (manager, admin API)
/// can discover what the active backend can do without probing individual
/// operations. Marked `#[non_exhaustive]` so new capability flags can be
/// added without breaking downstream code; construct values through
/// [`BackendCapabilities::minimal`] and the `with_*` builders.
#[non_exhaustive]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct BackendCapabilities {
/// Direct encryption of caller-provided plaintext with a master key
pub encrypt: bool,
/// Decryption of previously produced ciphertext
pub decrypt: bool,
/// Data encryption key (DEK) generation
pub generate_data_key: bool,
/// Key rotation that retains prior versions for decryption
pub rotate: bool,
/// Enabling and disabling keys
pub enable_disable: bool,
/// Scheduling key deletion with a pending window
pub schedule_deletion: bool,
/// Multiple key versions addressable after rotation
pub versioning: bool,
/// Irreversible physical deletion of key material
pub physical_delete: bool,
}
impl BackendCapabilities {
/// Conservative baseline: only the operations that every [`KmsBackend`]
/// implementation is required to provide by the trait. All optional
/// lifecycle capabilities default to unsupported.
pub const fn minimal() -> Self {
impl BackendInfo {
/// Create a new backend info
///
/// # Arguments
/// * `backend_type` - The type of the backend
/// * `version` - The version of the backend
/// * `endpoint` - The endpoint or location of the backend
/// * `healthy` - Whether the backend is healthy
///
/// # Returns
/// A new BackendInfo instance
///
pub fn new(backend_type: String, version: String, endpoint: String, healthy: bool) -> Self {
Self {
encrypt: true,
decrypt: true,
generate_data_key: true,
rotate: false,
enable_disable: false,
schedule_deletion: false,
versioning: false,
physical_delete: false,
backend_type,
version,
endpoint,
healthy,
metadata: HashMap::new(),
}
}
/// Set whether direct encryption is supported
pub const fn with_encrypt(mut self, encrypt: bool) -> Self {
self.encrypt = encrypt;
self
}
/// Set whether decryption is supported
pub const fn with_decrypt(mut self, decrypt: bool) -> Self {
self.decrypt = decrypt;
self
}
/// Set whether data key generation is supported
pub const fn with_generate_data_key(mut self, generate_data_key: bool) -> Self {
self.generate_data_key = generate_data_key;
self
}
/// Set whether version-retaining key rotation is supported
pub const fn with_rotate(mut self, rotate: bool) -> Self {
self.rotate = rotate;
self
}
/// Set whether enabling/disabling keys is supported
pub const fn with_enable_disable(mut self, enable_disable: bool) -> Self {
self.enable_disable = enable_disable;
self
}
/// Set whether scheduled deletion with a pending window is supported
pub const fn with_schedule_deletion(mut self, schedule_deletion: bool) -> Self {
self.schedule_deletion = schedule_deletion;
self
}
/// Set whether multiple key versions are supported
pub const fn with_versioning(mut self, versioning: bool) -> Self {
self.versioning = versioning;
self
}
/// Set whether physical deletion of key material is supported
pub const fn with_physical_delete(mut self, physical_delete: bool) -> Self {
self.physical_delete = physical_delete;
/// Add metadata to the backend info
///
/// # Arguments
/// * `key` - Metadata key
/// * `value` - Metadata value
///
/// # Returns
/// Updated BackendInfo instance
///
pub fn with_metadata(mut self, key: String, value: String) -> Self {
self.metadata.insert(key, value);
self
}
}
impl Default for BackendCapabilities {
fn default() -> Self {
Self::minimal()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::KmsConfig;
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
/// Backend that implements only the trait-mandated operations and relies
/// on the default `capabilities` implementation.
struct MinimalBackend;
#[async_trait]
impl KmsBackend for MinimalBackend {
async fn create_key(&self, _request: CreateKeyRequest) -> Result<CreateKeyResponse> {
unimplemented!("not exercised by capability tests")
}
async fn encrypt(&self, _request: EncryptRequest) -> Result<EncryptResponse> {
unimplemented!("not exercised by capability tests")
}
async fn decrypt(&self, _request: DecryptRequest) -> Result<DecryptResponse> {
unimplemented!("not exercised by capability tests")
}
async fn generate_data_key(&self, _request: GenerateDataKeyRequest) -> Result<GenerateDataKeyResponse> {
unimplemented!("not exercised by capability tests")
}
async fn describe_key(&self, _request: DescribeKeyRequest) -> Result<DescribeKeyResponse> {
unimplemented!("not exercised by capability tests")
}
async fn list_keys(&self, _request: ListKeysRequest) -> Result<ListKeysResponse> {
unimplemented!("not exercised by capability tests")
}
async fn delete_key(&self, _request: DeleteKeyRequest) -> Result<DeleteKeyResponse> {
unimplemented!("not exercised by capability tests")
}
async fn cancel_key_deletion(&self, _request: CancelKeyDeletionRequest) -> Result<CancelKeyDeletionResponse> {
unimplemented!("not exercised by capability tests")
}
async fn health_check(&self) -> Result<bool> {
Ok(true)
}
}
fn capabilities_snapshot(capabilities: BackendCapabilities) -> std::collections::BTreeMap<String, bool> {
serde_json::from_value(serde_json::to_value(capabilities).expect("capabilities should serialize"))
.expect("capabilities should deserialize into a flat bool map")
}
#[test]
fn default_capabilities_are_conservative() {
let capabilities = MinimalBackend.capabilities();
assert_eq!(capabilities, BackendCapabilities::minimal());
assert_eq!(capabilities, BackendCapabilities::default());
// The conservative baseline advertises only trait-mandated operations.
assert!(capabilities.encrypt);
assert!(capabilities.decrypt);
assert!(capabilities.generate_data_key);
assert!(!capabilities.rotate);
assert!(!capabilities.enable_disable);
assert!(!capabilities.schedule_deletion);
assert!(!capabilities.versioning);
assert!(!capabilities.physical_delete);
}
#[tokio::test]
async fn default_lifecycle_operations_are_unsupported() {
for (operation, result) in [
("enable_key", MinimalBackend.enable_key("any-key").await),
("disable_key", MinimalBackend.disable_key("any-key").await),
("rotate_key", MinimalBackend.rotate_key("any-key").await),
] {
let error = result.expect_err("backends must opt in to lifecycle operations by overriding them");
assert!(
matches!(error, KmsError::UnsupportedCapability { .. }),
"expected UnsupportedCapability for {operation}, got {error:?}"
);
}
}
#[tokio::test]
async fn default_remove_expired_key_is_unsupported() {
let error = MinimalBackend
.remove_expired_key("any-key", &jiff::Zoned::now())
.await
.expect_err("backends without deletion support must reject expired-key removal");
assert!(
matches!(error, KmsError::UnsupportedCapability { .. }),
"expected UnsupportedCapability, got {error:?}"
);
}
#[tokio::test]
async fn local_backend_capabilities_golden() {
let temp_dir = tempfile::tempdir().expect("temp dir should be created");
let config = KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults();
let backend = local::LocalKmsBackend::new(config).await.expect("local backend should build");
insta::assert_json_snapshot!("local_backend_capabilities", capabilities_snapshot(backend.capabilities()));
}
#[tokio::test]
async fn vault_kv2_backend_capabilities_golden() {
let config = KmsConfig::vault(
url::Url::parse("http://127.0.0.1:8200").expect("vault URL should parse"),
"dev-token".to_string(),
)
.with_insecure_development_defaults();
// Constructing the client performs no network I/O with token auth.
let backend = vault::VaultKmsBackend::new(config)
.await
.expect("vault kv2 backend should build");
insta::assert_json_snapshot!("vault_kv2_backend_capabilities", capabilities_snapshot(backend.capabilities()));
}
#[tokio::test]
async fn vault_transit_backend_capabilities_golden() {
let config = KmsConfig::vault_transit(
url::Url::parse("http://127.0.0.1:8200").expect("vault URL should parse"),
"dev-token".to_string(),
)
.with_insecure_development_defaults();
// Constructing the client performs no network I/O with token auth.
let backend = vault_transit::VaultTransitKmsBackend::new(config)
.await
.expect("vault transit backend should build");
insta::assert_json_snapshot!("vault_transit_backend_capabilities", capabilities_snapshot(backend.capabilities()));
}
#[tokio::test]
async fn static_backend_capabilities_golden() {
let config = KmsConfig::static_kms("static-key".to_string(), BASE64.encode([0u8; 32]));
let backend = static_kms::StaticKmsBackend::new(config)
.await
.expect("static backend should build");
insta::assert_json_snapshot!("static_backend_capabilities", capabilities_snapshot(backend.capabilities()));
}
}
-159
View File
@@ -1,159 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Minimal scripted HTTP responder standing in for a Vault server.
//!
//! Wiring tests need to observe how many Vault requests a code path performs
//! (retries, read-confirm recovery) without a live Vault. The responder serves
//! one canned response per incoming request in order, closes the connection
//! after each response, and records the `METHOD /path` sequence for
//! assertions. It intentionally implements just enough HTTP/1.1 for the
//! `vaultrs` reqwest client: no keep-alive, no chunked bodies.
use std::sync::{Arc, Mutex};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
/// One canned HTTP response.
pub(crate) struct ScriptedResponse {
status: u16,
body: String,
}
impl ScriptedResponse {
/// A 200 response carrying `data` inside the standard Vault envelope.
pub(crate) fn ok(data: serde_json::Value) -> Self {
Self {
status: 200,
body: serde_json::json!({
"request_id": "scripted",
"lease_id": "",
"lease_duration": 0,
"renewable": false,
"data": data,
})
.to_string(),
}
}
/// An error response in Vault's `{"errors": [...]}` format.
pub(crate) fn error(status: u16, message: &str) -> Self {
Self {
status,
body: serde_json::json!({ "errors": [message] }).to_string(),
}
}
}
/// A scripted stand-in Vault listening on a loopback port.
pub(crate) struct ScriptedVault {
/// Base address (`http://127.0.0.1:port`) to point a Vault client at.
pub(crate) address: String,
requests: Arc<Mutex<Vec<String>>>,
}
impl ScriptedVault {
/// Bind a loopback listener and serve `responses` one per request.
///
/// Requests beyond the script get a 599 error so a test that under-scripts
/// fails loudly instead of hanging.
pub(crate) async fn serve(responses: Vec<ScriptedResponse>) -> Self {
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind scripted vault listener");
let address = format!("http://{}", listener.local_addr().expect("scripted vault local addr"));
let requests = Arc::new(Mutex::new(Vec::new()));
let recorded = Arc::clone(&requests);
tokio::spawn(async move {
let mut responses = responses.into_iter();
loop {
let Ok((mut stream, _)) = listener.accept().await else {
return;
};
let Some(request_line) = read_request(&mut stream).await else {
continue;
};
recorded
.lock()
.expect("scripted vault request log poisoned")
.push(request_line);
let response = responses
.next()
.unwrap_or_else(|| ScriptedResponse::error(599, "scripted vault: script exhausted"));
let payload = format!(
"HTTP/1.1 {} Scripted\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{}",
response.status,
response.body.len(),
response.body
);
let _ = stream.write_all(payload.as_bytes()).await;
let _ = stream.shutdown().await;
}
});
Self { address, requests }
}
/// The `METHOD /path` lines of every request served so far, in order.
pub(crate) fn requests(&self) -> Vec<String> {
self.requests.lock().expect("scripted vault request log poisoned").clone()
}
}
/// Read one HTTP/1.1 request (head plus content-length body) and return its
/// `METHOD /path` line. Draining the body before responding keeps the client
/// from seeing a connection reset while it is still writing.
async fn read_request(stream: &mut TcpStream) -> Option<String> {
let mut buffer = Vec::new();
let mut chunk = [0u8; 4096];
let head_end = loop {
if let Some(position) = buffer.windows(4).position(|window| window == b"\r\n\r\n") {
break position + 4;
}
let read = stream.read(&mut chunk).await.ok()?;
if read == 0 {
return None;
}
buffer.extend_from_slice(&chunk[..read]);
};
let head = String::from_utf8_lossy(&buffer[..head_end]).into_owned();
let mut lines = head.lines();
let request_line = lines.next()?;
let mut parts = request_line.split_whitespace();
let method = parts.next()?;
let path = parts.next()?;
// rustify appends a lone "?" when an endpoint has no query parameters;
// strip it so assertions can use the plain path.
let path = path.strip_suffix('?').unwrap_or(path);
let content_length: usize = lines
.filter_map(|line| {
let (name, value) = line.split_once(':')?;
name.eq_ignore_ascii_case("content-length")
.then(|| value.trim().parse().ok())?
})
.next()
.unwrap_or(0);
let mut remaining = content_length.saturating_sub(buffer.len() - head_end);
while remaining > 0 {
let read = stream.read(&mut chunk).await.ok()?;
if read == 0 {
break;
}
remaining = remaining.saturating_sub(read);
}
Some(format!("{method} {path}"))
}
@@ -1,14 +0,0 @@
---
source: crates/kms/src/backends/mod.rs
expression: capabilities_snapshot(backend.capabilities())
---
{
"decrypt": true,
"enable_disable": true,
"encrypt": true,
"generate_data_key": true,
"physical_delete": true,
"rotate": false,
"schedule_deletion": true,
"versioning": false
}
@@ -1,14 +0,0 @@
---
source: crates/kms/src/backends/mod.rs
expression: capabilities_snapshot(backend.capabilities())
---
{
"decrypt": true,
"enable_disable": false,
"encrypt": true,
"generate_data_key": true,
"physical_delete": false,
"rotate": false,
"schedule_deletion": false,
"versioning": false
}
@@ -1,14 +0,0 @@
---
source: crates/kms/src/backends/mod.rs
expression: capabilities_snapshot(backend.capabilities())
---
{
"decrypt": true,
"enable_disable": true,
"encrypt": true,
"generate_data_key": true,
"physical_delete": true,
"rotate": true,
"schedule_deletion": true,
"versioning": true
}
@@ -1,14 +0,0 @@
---
source: crates/kms/src/backends/mod.rs
expression: capabilities_snapshot(backend.capabilities())
---
{
"decrypt": true,
"enable_disable": true,
"encrypt": true,
"generate_data_key": true,
"physical_delete": true,
"rotate": true,
"schedule_deletion": true,
"versioning": true
}
+142 -87
View File
@@ -21,7 +21,7 @@
//!
//! encrypted_data(plaintext_len+16) || nonce (12 bytes)
use crate::backends::{BackendCapabilities, KmsBackend};
use crate::backends::{BackendInfo, KmsBackend, KmsClient};
use crate::config::{BackendConfig, KmsConfig};
use crate::encryption::DataKeyEnvelope;
use crate::error::{KmsError, Result};
@@ -98,10 +98,9 @@ impl StaticKmsBackend {
}
}
impl StaticKmsBackend {
/// Generate a fresh data key and wrap it in the standard KMS envelope,
/// authenticated against the canonical encryption context.
pub(crate) fn generate_data_key_envelope(&self, request: &GenerateKeyRequest) -> Result<DataKeyInfo> {
#[async_trait]
impl KmsClient for StaticKmsBackend {
async fn generate_data_key(&self, request: &GenerateKeyRequest, _context: Option<&OperationContext>) -> Result<DataKeyInfo> {
if request.master_key_id != self.key_id {
return Err(KmsError::key_not_found(&request.master_key_id));
}
@@ -138,8 +137,6 @@ impl StaticKmsBackend {
nonce: nonce_bytes.to_vec(),
encryption_context: request.encryption_context.clone(),
created_at: Zoned::now(),
// The static backend has a single fixed key with no rotation.
master_key_version: None,
};
let ciphertext = serde_json::to_vec(&envelope)?;
@@ -152,8 +149,7 @@ impl StaticKmsBackend {
))
}
/// Encrypt caller-provided plaintext into the standard KMS envelope.
pub(crate) fn encrypt_to_envelope(&self, request: &EncryptRequest) -> Result<EncryptResponse> {
async fn encrypt(&self, request: &EncryptRequest, _context: Option<&OperationContext>) -> Result<EncryptResponse> {
if request.key_id != self.key_id {
return Err(KmsError::key_not_found(&request.key_id));
}
@@ -185,8 +181,6 @@ impl StaticKmsBackend {
nonce: nonce_bytes.to_vec(),
encryption_context: request.encryption_context.clone(),
created_at: Zoned::now(),
// The static backend has a single fixed key with no rotation.
master_key_version: None,
};
let ciphertext = serde_json::to_vec(&envelope)?;
@@ -198,8 +192,7 @@ impl StaticKmsBackend {
})
}
/// Open a KMS envelope produced by this backend.
pub(crate) fn decrypt_envelope(&self, request: &DecryptRequest) -> Result<Vec<u8>> {
async fn decrypt(&self, request: &DecryptRequest, _context: Option<&OperationContext>) -> Result<Vec<u8>> {
let envelope: DataKeyEnvelope = serde_json::from_slice(&request.ciphertext)
.map_err(|error| KmsError::cryptographic_error("parse", format!("Failed to parse data key envelope: {error}")))?;
if envelope.master_key_id != self.key_id {
@@ -238,8 +231,14 @@ impl StaticKmsBackend {
Ok(plaintext)
}
/// Describe the single configured key.
pub(crate) fn configured_key_info(&self, key_id: &str) -> Result<KeyInfo> {
async fn create_key(&self, key_id: &str, _algorithm: &str, _context: Option<&OperationContext>) -> Result<MasterKeyInfo> {
if key_id == self.key_id {
return Err(KmsError::key_already_exists(key_id));
}
Err(KmsError::invalid_operation("Static KMS is read-only: cannot create new keys"))
}
async fn describe_key(&self, key_id: &str, _context: Option<&OperationContext>) -> Result<KeyInfo> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
@@ -260,8 +259,7 @@ impl StaticKmsBackend {
})
}
/// List the single configured key, honouring the pagination marker.
pub(crate) fn list_configured_key(&self, request: &ListKeysRequest) -> Result<ListKeysResponse> {
async fn list_keys(&self, request: &ListKeysRequest, _context: Option<&OperationContext>) -> Result<ListKeysResponse> {
let key_info = KeyInfo {
key_id: self.key_id.clone(),
description: Some("Static single-key KMS backend".to_string()),
@@ -293,6 +291,57 @@ impl StaticKmsBackend {
truncated: false,
})
}
async fn enable_key(&self, key_id: &str, _context: Option<&OperationContext>) -> Result<()> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
// Static KMS key is always enabled
Ok(())
}
async fn disable_key(&self, key_id: &str, _context: Option<&OperationContext>) -> Result<()> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
Err(KmsError::invalid_operation("Static KMS is read-only: cannot disable keys"))
}
async fn schedule_key_deletion(
&self,
key_id: &str,
_pending_window_days: u32,
_context: Option<&OperationContext>,
) -> Result<()> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
Err(KmsError::invalid_operation("Static KMS is read-only: cannot schedule key deletion"))
}
async fn cancel_key_deletion(&self, key_id: &str, _context: Option<&OperationContext>) -> Result<()> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
Err(KmsError::invalid_operation("Static KMS is read-only: cannot cancel key deletion"))
}
async fn rotate_key(&self, key_id: &str, _context: Option<&OperationContext>) -> Result<MasterKeyInfo> {
if key_id != self.key_id {
return Err(KmsError::key_not_found(key_id));
}
Err(KmsError::invalid_operation("Static KMS is read-only: cannot rotate keys"))
}
async fn health_check(&self) -> Result<()> {
// Static KMS is always healthy if it was successfully initialized
Ok(())
}
fn backend_info(&self) -> BackendInfo {
BackendInfo::new("static".to_string(), env!("CARGO_PKG_VERSION").to_string(), "local".to_string(), true)
.with_metadata("key_id".to_string(), self.key_id.clone())
}
}
#[async_trait]
@@ -306,12 +355,12 @@ impl KmsBackend for StaticKmsBackend {
}
async fn encrypt(&self, request: EncryptRequest) -> Result<EncryptResponse> {
self.encrypt_to_envelope(&request)
<Self as KmsClient>::encrypt(self, &request, None).await
}
async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse> {
let key_id = self.key_id.clone();
let plaintext = self.decrypt_envelope(&request)?;
let plaintext = <Self as KmsClient>::decrypt(self, &request, None).await?;
Ok(DecryptResponse {
plaintext,
key_id,
@@ -327,7 +376,7 @@ impl KmsBackend for StaticKmsBackend {
encryption_context: request.encryption_context,
grant_tokens: Vec::new(),
};
let data_key = self.generate_data_key_envelope(&gen_req)?;
let data_key = <Self as KmsClient>::generate_data_key(self, &gen_req, None).await?;
let plaintext_key = data_key
.plaintext
@@ -342,7 +391,7 @@ impl KmsBackend for StaticKmsBackend {
}
async fn describe_key(&self, request: DescribeKeyRequest) -> Result<DescribeKeyResponse> {
let key_info = self.configured_key_info(&request.key_id)?;
let key_info = <Self as KmsClient>::describe_key(self, &request.key_id, None).await?;
let key_metadata = KeyMetadata {
key_id: key_info.key_id.clone(),
key_state: if key_info.status == KeyStatus::Active {
@@ -362,7 +411,7 @@ impl KmsBackend for StaticKmsBackend {
}
async fn list_keys(&self, request: ListKeysRequest) -> Result<ListKeysResponse> {
self.list_configured_key(&request)
<Self as KmsClient>::list_keys(self, &request, None).await
}
async fn delete_key(&self, request: DeleteKeyRequest) -> Result<DeleteKeyResponse> {
@@ -382,18 +431,12 @@ impl KmsBackend for StaticKmsBackend {
async fn health_check(&self) -> Result<bool> {
Ok(true)
}
fn capabilities(&self) -> BackendCapabilities {
// Static KMS is a read-only single-key backend: it only performs
// cryptographic operations and rejects every lifecycle mutation.
BackendCapabilities::minimal()
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::backends::KmsBackend as KmsBackendTrait;
use crate::backends::{KmsBackend as KmsBackendTrait, KmsClient};
use crate::config::{BackendConfig, KmsBackend, StaticConfig};
use crate::encryption::is_data_key_envelope;
use base64::Engine as _;
@@ -438,8 +481,8 @@ mod tests {
// Generate data key
let request = GenerateKeyRequest::new(key_id.clone(), "AES_256".to_string())
.with_context("bucket".to_string(), "test-bucket".to_string());
let data_key = backend
.generate_data_key_envelope(&request)
let data_key = KmsClient::generate_data_key(&backend, &request, None)
.await
.expect("Failed to generate data key");
assert_eq!(data_key.key_id, key_id);
@@ -456,7 +499,9 @@ mod tests {
// Decrypt the data key
let decrypt_request =
DecryptRequest::new(data_key.ciphertext.clone()).with_context("bucket".to_string(), "test-bucket".to_string());
let decrypted = backend.decrypt_envelope(&decrypt_request).expect("Failed to decrypt");
let decrypted = KmsClient::decrypt(&backend, &decrypt_request, None)
.await
.expect("Failed to decrypt");
assert_eq!(decrypted.as_slice(), data_key.plaintext.as_deref().expect("plaintext should exist"));
}
@@ -468,8 +513,8 @@ mod tests {
.with_context("bucket".to_string(), "source-bucket".to_string())
.with_context("object".to_string(), "source-object".to_string());
let data_key = backend
.generate_data_key_envelope(&request)
let data_key = KmsClient::generate_data_key(&backend, &request, None)
.await
.expect("generate static KMS data key");
assert!(
@@ -513,8 +558,8 @@ mod tests {
let (backend, key_id, _key) = create_test_backend().await;
let request = GenerateKeyRequest::new(key_id, "AES_256".to_string())
.with_context("bucket".to_string(), "source-bucket".to_string());
let generated = backend
.generate_data_key_envelope(&request)
let generated = KmsClient::generate_data_key(&backend, &request, None)
.await
.expect("generate context-bound data key");
let mut envelope: DataKeyEnvelope = serde_json::from_slice(&generated.ciphertext).expect("parse static KMS envelope");
envelope
@@ -523,8 +568,8 @@ mod tests {
let decrypt_request = DecryptRequest::new(serde_json::to_vec(&envelope).expect("serialize tampered envelope"))
.with_context("bucket".to_string(), "different-bucket".to_string());
let error = backend
.decrypt_envelope(&decrypt_request)
let error = KmsClient::decrypt(&backend, &decrypt_request, None)
.await
.expect_err("tampering with authenticated envelope context must fail");
assert!(matches!(error, KmsError::CryptographicError { .. }));
@@ -535,7 +580,7 @@ mod tests {
let (backend, _key_id, _key) = create_test_backend().await;
let request = GenerateKeyRequest::new("wrong-key-id".to_string(), "AES_256".to_string());
let result = backend.generate_data_key_envelope(&request);
let result = KmsClient::generate_data_key(&backend, &request, None).await;
assert!(result.is_err());
assert!(result.expect_err("should be Err").to_string().contains("wrong-key-id"));
}
@@ -547,7 +592,7 @@ mod tests {
// Ciphertext too short
let short = vec![0u8; 10];
let request = DecryptRequest::new(short);
let result = backend.decrypt_envelope(&request);
let result = KmsClient::decrypt(&backend, &request, None).await;
assert!(result.is_err());
}
@@ -557,7 +602,9 @@ mod tests {
// Generate a valid ciphertext first
let gen_request = GenerateKeyRequest::new(key_id, "AES_256".to_string());
let data_key = backend.generate_data_key_envelope(&gen_request).expect("generate");
let data_key = KmsClient::generate_data_key(&backend, &gen_request, None)
.await
.expect("generate");
// Tamper with the ciphertext (flip a bit in the encrypted portion)
let mut tampered = data_key.ciphertext.clone();
@@ -566,7 +613,7 @@ mod tests {
}
let request = DecryptRequest::new(tampered);
let result = backend.decrypt_envelope(&request);
let result = KmsClient::decrypt(&backend, &request, None).await;
assert!(result.is_err());
}
@@ -575,14 +622,7 @@ mod tests {
let (backend, key_id, _key) = create_test_backend().await;
// Creating the pre-configured key should return KeyAlreadyExists
let result = KmsBackendTrait::create_key(
&backend,
CreateKeyRequest {
key_name: Some(key_id.clone()),
..Default::default()
},
)
.await;
let result = KmsClient::create_key(&backend, &key_id, "AES_256", None).await;
assert!(result.is_err());
assert!(result.expect_err("should be Err").to_string().contains("already exists"));
}
@@ -592,14 +632,7 @@ mod tests {
let (backend, _key_id, _key) = create_test_backend().await;
// Creating any other key should return invalid operation (read-only)
let result = KmsBackendTrait::create_key(
&backend,
CreateKeyRequest {
key_name: Some("other-key".to_string()),
..Default::default()
},
)
.await;
let result = KmsClient::create_key(&backend, "other-key", "AES_256", None).await;
assert!(result.is_err());
let err_msg = result.expect_err("should be Err").to_string();
assert!(err_msg.contains("read-only") || err_msg.contains("cannot create"));
@@ -609,13 +642,15 @@ mod tests {
async fn test_describe_key() {
let (backend, key_id, _key) = create_test_backend().await;
let key_info = backend.configured_key_info(&key_id).expect("describe_key should succeed");
let key_info = KmsClient::describe_key(&backend, &key_id, None)
.await
.expect("describe_key should succeed");
assert_eq!(key_info.key_id, key_id);
assert_eq!(key_info.status, KeyStatus::Active);
assert_eq!(key_info.algorithm, "AES_256");
// Wrong key ID
let result = backend.configured_key_info("nonexistent");
let result = KmsClient::describe_key(&backend, "nonexistent", None).await;
assert!(result.is_err());
}
@@ -623,8 +658,8 @@ mod tests {
async fn test_list_keys() {
let (backend, key_id, _key) = create_test_backend().await;
let response = backend
.list_configured_key(&ListKeysRequest::default())
let response = KmsClient::list_keys(&backend, &ListKeysRequest::default(), None)
.await
.expect("list_keys should succeed");
assert_eq!(response.keys.len(), 1);
assert_eq!(response.keys[0].key_id, key_id);
@@ -632,45 +667,61 @@ mod tests {
}
#[tokio::test]
async fn lifecycle_mutations_are_unsupported_at_the_product_surface() {
async fn test_disable_key_returns_error() {
let (backend, key_id, _key) = create_test_backend().await;
// The static backend advertises no enable/disable or rotation
// capability, so the shared KmsBackend defaults reject all three.
for result in [
KmsBackendTrait::enable_key(&backend, &key_id).await,
KmsBackendTrait::disable_key(&backend, &key_id).await,
KmsBackendTrait::rotate_key(&backend, &key_id).await,
] {
let error = result.expect_err("static lifecycle mutations must be rejected");
assert!(matches!(error, KmsError::UnsupportedCapability { .. }), "got {error:?}");
}
let result = KmsClient::disable_key(&backend, &key_id, None).await;
assert!(result.is_err());
assert!(result.expect_err("should be Err").to_string().contains("read-only"));
}
#[tokio::test]
async fn test_enable_key_is_noop() {
let (backend, key_id, _key) = create_test_backend().await;
// Enable should succeed (no-op for static KMS)
KmsClient::enable_key(&backend, &key_id, None)
.await
.expect("enable_key should be no-op");
// Wrong key should still fail
let result = KmsClient::enable_key(&backend, "wrong", None).await;
assert!(result.is_err());
}
#[tokio::test]
async fn test_delete_key_returns_error() {
let (backend, key_id, _key) = create_test_backend().await;
let result = KmsBackendTrait::delete_key(
&backend,
DeleteKeyRequest {
key_id: key_id.clone(),
pending_window_in_days: Some(7),
force_immediate: None,
},
)
.await;
let result = KmsClient::schedule_key_deletion(&backend, &key_id, 7, None).await;
assert!(result.is_err());
assert!(result.expect_err("should be Err").to_string().contains("read-only"));
}
#[tokio::test]
async fn test_rotate_key_returns_error() {
let (backend, key_id, _key) = create_test_backend().await;
let result = KmsClient::rotate_key(&backend, &key_id, None).await;
assert!(result.is_err());
}
#[tokio::test]
async fn test_health_check() {
let (backend, _key_id, _key) = create_test_backend().await;
KmsBackendTrait::health_check(&backend)
.await
.expect("health_check should succeed");
KmsClient::health_check(&backend).await.expect("health_check should succeed");
}
#[tokio::test]
async fn test_backend_info() {
let (backend, key_id, _key) = create_test_backend().await;
let info = KmsClient::backend_info(&backend);
assert_eq!(info.backend_type, "static");
assert_eq!(info.endpoint, "local");
assert!(info.healthy);
assert_eq!(info.metadata.get("key_id"), Some(&key_id));
}
#[tokio::test]
@@ -679,13 +730,17 @@ mod tests {
let plaintext = b"Hello, static KMS world!";
let enc_request = EncryptRequest::new(key_id.clone(), plaintext.to_vec());
let enc_response = backend.encrypt_to_envelope(&enc_request).expect("encrypt should succeed");
let enc_response = KmsClient::encrypt(&backend, &enc_request, None)
.await
.expect("encrypt should succeed");
assert_eq!(enc_response.key_id, key_id);
assert!(!enc_response.ciphertext.is_empty());
let dec_request = DecryptRequest::new(enc_response.ciphertext);
let decrypted = backend.decrypt_envelope(&dec_request).expect("decrypt should succeed");
let decrypted = KmsClient::decrypt(&backend, &dec_request, None)
.await
.expect("decrypt should succeed");
assert_eq!(decrypted, plaintext);
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-233
View File
@@ -1,233 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Backup responsibility matrix: what a RustFS backup bundle owns per backend.
//!
//! The matrix is two-dimensional on purpose: responsibility is a function of
//! the backend *and* its at-rest protection state, not of the backend alone.
//! This keeps the schema stable when a backend changes protection direction —
//! switching Vault KV2 between storage-only and Transit-wrapped operation
//! selects a different existing row instead of changing the contract.
//!
//! These enums are backup-domain contract types. Once the backlog#1571
//! capability-discovery contract lands, the discovery surface is expected to
//! converge on (or map onto) the states defined here.
use serde::{Deserialize, Serialize};
/// Backend discriminant recorded in a backup manifest.
///
/// Wire names are aligned with [`crate::config::BackendConfig`] and
/// [`crate::config::KmsBackend`] (including the legacy `Vault` alias) so that
/// a manifest and a persisted KMS configuration never disagree about how the
/// same backend is spelled.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum BackupBackendKind {
/// Local file-based backend.
Local,
/// Vault KV v2 storage backend.
#[serde(rename = "VaultKV2", alias = "Vault")]
VaultKv2,
/// Vault Transit backend.
VaultTransit,
/// Static single-key backend.
Static,
}
/// At-rest protection state of master key material, as observed at snapshot
/// time.
///
/// The first three states mirror the Local backend's on-disk protection
/// marker (`StoredKeyProtection` in `backends/local.rs`, kebab-case wire
/// names). The remaining states describe the non-local backends.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum AtRestProtection {
/// Local key files AEAD-encrypted under the Argon2id-derived master key.
EncryptedMasterKey,
/// Local development-only plaintext key files. Such files must never
/// enter a bundle as-is; bundle artifacts are always re-wrapped under the
/// backup KEK.
PlaintextDevOnly,
/// Pre-beta.9 local key files without a protection marker; the effective
/// mode is resolved at read time. Treated like [`Self::PlaintextDevOnly`]
/// for bundling purposes: re-wrap is mandatory.
LegacyUnspecified,
/// Vault KV2 as currently shipped: material confidentiality relies on
/// Vault ACLs, KV2 at-rest encryption, and TLS only (the backend reports
/// `at_rest_protection = "vault-kv2-acl"`); RustFS applies no
/// cryptographic wrapping of its own.
StorageOnly,
/// Vault KV2 with material wrapped by Vault Transit before storage. Not
/// produced by any current backend; the row exists so a future direction
/// change selects a state instead of changing the schema.
TransitWrapped,
/// Vault Transit: the cryptographic root lives in Vault and is not
/// exportable. RustFS can only ever own metadata and references.
ExternalNonExportable,
/// Static backend: the secret is delivered externally at startup and
/// RustFS persists no key material at all.
ExternalSecretDelivery,
}
/// What a RustFS backup bundle is responsible for, per (backend, protection)
/// combination.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum BackupResponsibility {
/// The bundle carries the complete recoverable state: encrypted key
/// material for every version, salt, metadata, and configuration.
///
/// Restore precondition for the Local backend: the operator re-supplies
/// the master key out of band. The master key itself is outside the
/// backup domain; the manifest stores at most an opaque verifier.
FullMaterial,
/// The bundle carries only non-sensitive references and verification
/// information. The source of truth is external secret delivery and the
/// operator re-provides the secret during restore. Embedding the secret
/// itself in a bundle is forbidden.
ReferenceOnly,
/// The bundle carries RustFS-side metadata, configuration references and
/// verification data, while the cryptographic root is protected by the
/// external system's native snapshot/disaster-recovery flow (Vault/HSM).
/// Restore must re-establish the external trust root first.
MetadataPlusExternalRoot,
}
impl BackupResponsibility {
/// Resolve the responsibility matrix for one (backend, protection) cell.
///
/// Returns `None` for combinations that no supported deployment can
/// produce; manifests declaring such a combination are rejected as
/// corrupted. This function is total and the unit tests anchor every
/// cell, so any change to the matrix is a deliberate contract change.
pub fn for_backend(backend: BackupBackendKind, protection: AtRestProtection) -> Option<Self> {
use AtRestProtection::*;
use BackupBackendKind::*;
match (backend, protection) {
(Local, EncryptedMasterKey | PlaintextDevOnly | LegacyUnspecified) => Some(Self::FullMaterial),
(Local, _) => None,
// Storage-only KV2 offers no external cryptographic root, so the
// bundle must own the material (re-wrapped under the backup KEK).
(VaultKv2, StorageOnly) => Some(Self::FullMaterial),
(VaultKv2, TransitWrapped) => Some(Self::MetadataPlusExternalRoot),
(VaultKv2, _) => None,
(VaultTransit, ExternalNonExportable) => Some(Self::MetadataPlusExternalRoot),
(VaultTransit, _) => None,
(Static, ExternalSecretDelivery) => Some(Self::ReferenceOnly),
(Static, _) => None,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::KmsBackend;
fn json<T: Serialize>(value: &T) -> String {
serde_json::to_string(value).expect("serialization should succeed")
}
#[test]
fn backend_kind_wire_names_match_kms_backend() {
let pairs = [
(BackupBackendKind::Local, KmsBackend::Local),
(BackupBackendKind::VaultKv2, KmsBackend::VaultKv2),
(BackupBackendKind::VaultTransit, KmsBackend::VaultTransit),
(BackupBackendKind::Static, KmsBackend::Static),
];
for (backup_kind, config_kind) in pairs {
assert_eq!(json(&backup_kind), json(&config_kind), "wire name drifted for {backup_kind:?}");
}
}
#[test]
fn backend_kind_accepts_legacy_vault_alias() {
let decoded: BackupBackendKind = serde_json::from_str("\"Vault\"").expect("legacy alias should decode");
assert_eq!(decoded, BackupBackendKind::VaultKv2);
}
#[test]
fn responsibility_matrix_is_anchored_cell_by_cell() {
use AtRestProtection::*;
use BackupBackendKind::*;
use BackupResponsibility::*;
// Every (backend, protection) cell, exhaustively. Changing any row is
// a contract change and must be made here consciously.
let matrix = [
(Local, EncryptedMasterKey, Some(FullMaterial)),
(Local, PlaintextDevOnly, Some(FullMaterial)),
(Local, LegacyUnspecified, Some(FullMaterial)),
(Local, StorageOnly, None),
(Local, TransitWrapped, None),
(Local, ExternalNonExportable, None),
(Local, ExternalSecretDelivery, None),
(VaultKv2, EncryptedMasterKey, None),
(VaultKv2, PlaintextDevOnly, None),
(VaultKv2, LegacyUnspecified, None),
(VaultKv2, StorageOnly, Some(FullMaterial)),
(VaultKv2, TransitWrapped, Some(MetadataPlusExternalRoot)),
(VaultKv2, ExternalNonExportable, None),
(VaultKv2, ExternalSecretDelivery, None),
(VaultTransit, EncryptedMasterKey, None),
(VaultTransit, PlaintextDevOnly, None),
(VaultTransit, LegacyUnspecified, None),
(VaultTransit, StorageOnly, None),
(VaultTransit, TransitWrapped, None),
(VaultTransit, ExternalNonExportable, Some(MetadataPlusExternalRoot)),
(VaultTransit, ExternalSecretDelivery, None),
(Static, EncryptedMasterKey, None),
(Static, PlaintextDevOnly, None),
(Static, LegacyUnspecified, None),
(Static, StorageOnly, None),
(Static, TransitWrapped, None),
(Static, ExternalNonExportable, None),
(Static, ExternalSecretDelivery, Some(ReferenceOnly)),
];
assert_eq!(matrix.len(), 28, "matrix must stay exhaustive: 4 backends x 7 protection states");
for (backend, protection, expected) in matrix {
assert_eq!(
BackupResponsibility::for_backend(backend, protection),
expected,
"matrix cell drifted for ({backend:?}, {protection:?})"
);
}
}
#[test]
fn local_protection_wire_names_match_stored_key_protection() {
use crate::backends::local::StoredKeyProtection;
// The manifest must record exactly the marker values the Local
// backend writes to disk, or a restore could misread protection.
let pairs = [
(AtRestProtection::EncryptedMasterKey, StoredKeyProtection::EncryptedMasterKey),
(AtRestProtection::PlaintextDevOnly, StoredKeyProtection::PlaintextDevOnly),
(AtRestProtection::LegacyUnspecified, StoredKeyProtection::LegacyUnspecified),
];
for (backup_state, stored_state) in pairs {
assert_eq!(json(&backup_state), json(&stored_state), "wire name drifted for {backup_state:?}");
}
}
#[test]
fn responsibility_wire_names_are_frozen() {
assert_eq!(json(&BackupResponsibility::FullMaterial), "\"full-material\"");
assert_eq!(json(&BackupResponsibility::ReferenceOnly), "\"reference-only\"");
assert_eq!(json(&BackupResponsibility::MetadataPlusExternalRoot), "\"metadata-plus-external-root\"");
}
}
-280
View File
@@ -1,280 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Restore dry-run report contract.
//!
//! A restore dry-run is a zero-write preflight: it evaluates a bundle against
//! a target and reports every blocker, conflict, and external dependency
//! mismatch without modifying the target in any way. The report itself is
//! plain data — producing, serializing, or discarding it has no side effects,
//! and an implementation that writes anything during a dry-run violates this
//! contract. All values in a report are identifiers and references; secrets,
//! tokens, and key material never appear in it.
use crate::backup::error::BackupError;
use serde::{Deserialize, Serialize};
/// Machine-readable category of a restore blocker.
///
/// The first six codes mirror the [`BackupError`] variants; the remaining
/// codes cover preflight conditions that are not bundle defects.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum RestoreBlockerCode {
/// The bundle failed structural or integrity validation.
BundleCorrupted,
/// The manifest input ended prematurely.
BundleTruncated,
/// The manifest format version is unknown to this build.
UnknownFormatVersion,
/// The supplied backup KEK does not match the bundle's KEK.
WrongBackupKek,
/// A required artifact is absent from the bundle.
MissingArtifact,
/// The bundle has no completeness marker or is marked in-progress.
IncompleteBundle,
/// The target backend cannot satisfy the bundle's responsibility model.
UnsupportedBackend,
/// The bundle was produced by a different deployment than the target and
/// no explicit cross-deployment authorization applies.
DeploymentMismatch,
/// An external dependency (Vault cluster, mount, Transit key, ...) that
/// the bundle references is unreachable or missing.
ExternalDependencyUnavailable,
}
/// One condition that forbids the restore outright.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RestoreBlocker {
/// Machine-readable category.
pub code: RestoreBlockerCode,
/// Human-readable detail. Identifiers only; never secrets or material.
pub detail: String,
}
impl From<&BackupError> for RestoreBlocker {
fn from(error: &BackupError) -> Self {
let code = match error {
BackupError::Corrupted { .. } => RestoreBlockerCode::BundleCorrupted,
BackupError::Truncated { .. } => RestoreBlockerCode::BundleTruncated,
BackupError::UnknownVersion { .. } => RestoreBlockerCode::UnknownFormatVersion,
BackupError::WrongKek { .. } => RestoreBlockerCode::WrongBackupKek,
BackupError::MissingArtifact { .. } => RestoreBlockerCode::MissingArtifact,
BackupError::IncompleteBundle { .. } => RestoreBlockerCode::IncompleteBundle,
};
Self {
code,
detail: error.to_string(),
}
}
}
/// Kind of a conflict between bundle state and existing target state.
///
/// Restore is non-destructive by default: every conflict blocks the restore
/// unless an explicit, audited conflict policy resolves it. Silent overwrite
/// or merge is never an option.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum RestoreConflictKind {
/// The target already has a key with this stable id.
KeyAlreadyExists,
/// Restoring would lower a key version the target has already observed.
VersionRegression,
/// Restoring would lower the snapshot generation the target has already
/// observed.
GenerationRegression,
/// Restoring would revive a key the target has deleted or scheduled for
/// deletion.
StateRegression,
}
/// One conflict with existing target state.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RestoreConflict {
/// Stable key id the conflict concerns.
pub key_id: String,
/// Machine-readable category.
pub kind: RestoreConflictKind,
/// Human-readable detail. Identifiers only; never secrets or material.
pub detail: String,
}
/// A mismatch between an external dependency reference recorded in the bundle
/// and what the target environment observes.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ExternalDependencyMismatch {
/// Which dependency is affected (for example a Vault mount or Transit
/// key name). References only; never credentials.
pub dependency: String,
/// The value the bundle recorded.
pub expected: String,
/// The value the target environment reports.
pub observed: String,
}
/// Result of a restore dry-run preflight.
///
/// # Zero-write contract
///
/// A dry-run must not write to the target: no staging directories, no
/// repaired records, no metadata fixes triggered along the read path. The
/// report is pure data over values already known to the caller.
///
/// ```
/// use rustfs_kms::backup::{RestoreBlocker, RestoreBlockerCode, RestoreDryRunReport};
///
/// let clean = RestoreDryRunReport {
/// backup_id: "backup-0001".to_string(),
/// target_deployment_identity: "deployment-a".to_string(),
/// blockers: Vec::new(),
/// conflicts: Vec::new(),
/// external_mismatches: Vec::new(),
/// };
/// assert!(clean.restore_permitted());
///
/// let blocked = RestoreDryRunReport {
/// blockers: vec![RestoreBlocker {
/// code: RestoreBlockerCode::IncompleteBundle,
/// detail: "manifest has no completeness marker".to_string(),
/// }],
/// ..clean
/// };
/// assert!(!blocked.restore_permitted());
/// ```
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct RestoreDryRunReport {
/// Identifier of the evaluated bundle.
pub backup_id: String,
/// Identity of the restore target the bundle was evaluated against.
pub target_deployment_identity: String,
/// Conditions that forbid the restore outright.
pub blockers: Vec<RestoreBlocker>,
/// Conflicts with existing target state.
pub conflicts: Vec<RestoreConflict>,
/// External dependency mismatches.
pub external_mismatches: Vec<ExternalDependencyMismatch>,
}
impl RestoreDryRunReport {
/// Whether the restore may proceed: true only when the preflight found
/// no blockers, no conflicts, and no external dependency mismatches.
pub fn restore_permitted(&self) -> bool {
self.blockers.is_empty() && self.conflicts.is_empty() && self.external_mismatches.is_empty()
}
}
#[cfg(test)]
mod tests {
use super::*;
fn sample_report() -> RestoreDryRunReport {
RestoreDryRunReport {
backup_id: "backup-0001".to_string(),
target_deployment_identity: "deployment-b".to_string(),
blockers: vec![RestoreBlocker {
code: RestoreBlockerCode::DeploymentMismatch,
detail: "bundle was produced by deployment-a".to_string(),
}],
conflicts: vec![RestoreConflict {
key_id: "object-key".to_string(),
kind: RestoreConflictKind::VersionRegression,
detail: "target observed version 5, bundle carries version 3".to_string(),
}],
external_mismatches: vec![ExternalDependencyMismatch {
dependency: "vault transit key rustfs-master".to_string(),
expected: "min_version=2".to_string(),
observed: "min_version=4".to_string(),
}],
}
}
#[test]
fn report_round_trips_through_json() {
let report = sample_report();
let json = serde_json::to_string(&report).expect("serialization should succeed");
let decoded: RestoreDryRunReport = serde_json::from_str(&json).expect("deserialization should succeed");
assert_eq!(decoded, report);
}
#[test]
fn restore_permitted_requires_every_section_empty() {
assert!(!sample_report().restore_permitted());
let clean = RestoreDryRunReport {
blockers: Vec::new(),
conflicts: Vec::new(),
external_mismatches: Vec::new(),
..sample_report()
};
assert!(clean.restore_permitted());
for section in 0..3 {
let mut report = clean.clone();
match section {
0 => report.blockers = sample_report().blockers,
1 => report.conflicts = sample_report().conflicts,
_ => report.external_mismatches = sample_report().external_mismatches,
}
assert!(!report.restore_permitted(), "section {section} alone must block the restore");
}
}
#[test]
fn every_backup_error_maps_to_a_blocker_code() {
let cases = [
(BackupError::corrupted("x"), RestoreBlockerCode::BundleCorrupted),
(BackupError::truncated("x"), RestoreBlockerCode::BundleTruncated),
(
BackupError::UnknownVersion { found: 2, supported: 1 },
RestoreBlockerCode::UnknownFormatVersion,
),
(
BackupError::WrongKek {
required_kek_id: "a".to_string(),
required_kek_version: 1,
supplied_kek_id: "b".to_string(),
supplied_kek_version: 1,
},
RestoreBlockerCode::WrongBackupKek,
),
(BackupError::missing_artifact("key-material"), RestoreBlockerCode::MissingArtifact),
(BackupError::incomplete_bundle("x"), RestoreBlockerCode::IncompleteBundle),
];
for (error, expected_code) in cases {
let blocker = RestoreBlocker::from(&error);
assert_eq!(blocker.code, expected_code, "wrong code for {error:?}");
assert_eq!(blocker.detail, error.to_string());
}
}
/// The zero-write contract in practice: a report is plain serializable
/// data with no handles, no I/O, and no drop side effects.
#[test]
fn report_types_are_plain_data() {
fn assert_plain_data<T>()
where
T: serde::Serialize + serde::de::DeserializeOwned + Clone + PartialEq + std::fmt::Debug + Send + Sync + 'static,
{
}
assert_plain_data::<RestoreDryRunReport>();
assert_plain_data::<RestoreBlocker>();
assert_plain_data::<RestoreConflict>();
assert_plain_data::<ExternalDependencyMismatch>();
}
}
-133
View File
@@ -1,133 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Typed failures for the backup/restore bundle contract.
use thiserror::Error;
/// Typed failures raised while decoding or validating a backup bundle.
///
/// Every variant is a fail-closed condition: a restore surface observing any
/// of them must abort before touching target state. Messages carry only
/// identifiers (backup ids, KEK ids, artifact kinds and paths) — never key
/// material, bundle plaintext, or credentials.
#[derive(Error, Debug, Clone, PartialEq, Eq)]
pub enum BackupError {
/// Manifest or bundle content fails structural, schema, or integrity
/// validation (unknown fields, duplicate fields, digest mismatch,
/// contradictory responsibility declarations, ...).
#[error("backup bundle corrupted: {reason}")]
Corrupted { reason: String },
/// Input ended before a complete manifest could be decoded.
#[error("backup manifest truncated: {reason}")]
Truncated { reason: String },
/// Manifest declares a format version this build does not understand.
/// Unknown versions are always rejected; there is no best-effort read.
#[error("unknown backup manifest format version {found} (this build supports version {supported})")]
UnknownVersion { found: u32, supported: u32 },
/// Bundle is protected by a different backup KEK than the one supplied.
#[error(
"backup bundle requires KEK '{required_kek_id}' version {required_kek_version}; \
supplied KEK '{supplied_kek_id}' version {supplied_kek_version} cannot open it"
)]
WrongKek {
required_kek_id: String,
required_kek_version: u32,
supplied_kek_id: String,
supplied_kek_version: u32,
},
/// Manifest requires an artifact that is not present in the bundle.
#[error("backup bundle is missing a required artifact: {artifact}")]
MissingArtifact { artifact: String },
/// Bundle has no completeness marker or records an in-progress state.
/// A bundle that never reached its completeness marker must never be
/// restored, regardless of how much of it is readable.
#[error("backup bundle is incomplete ({reason}); incomplete bundles must never be restored")]
IncompleteBundle { reason: String },
}
impl BackupError {
/// Create a corrupted-bundle error.
pub fn corrupted<S: Into<String>>(reason: S) -> Self {
Self::Corrupted { reason: reason.into() }
}
/// Create a truncated-manifest error.
pub fn truncated<S: Into<String>>(reason: S) -> Self {
Self::Truncated { reason: reason.into() }
}
/// Create an incomplete-bundle error.
pub fn incomplete_bundle<S: Into<String>>(reason: S) -> Self {
Self::IncompleteBundle { reason: reason.into() }
}
/// Create a missing-artifact error.
pub fn missing_artifact<S: Into<String>>(artifact: S) -> Self {
Self::MissingArtifact {
artifact: artifact.into(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::error::KmsError;
#[test]
fn backup_errors_convert_into_kms_error_transparently() {
let error = BackupError::corrupted("manifest digest mismatch");
let kms_error: KmsError = error.clone().into();
assert_eq!(kms_error.to_string(), error.to_string());
assert!(matches!(kms_error, KmsError::Backup(inner) if inner == error));
}
#[test]
fn error_messages_carry_identifiers_only() {
// The display strings must stay descriptive without ever embedding
// material or bundle plaintext; each variant only interpolates the
// identifiers below.
let wrong_kek = BackupError::WrongKek {
required_kek_id: "backup-kek-1".to_string(),
required_kek_version: 3,
supplied_kek_id: "backup-kek-2".to_string(),
supplied_kek_version: 1,
};
assert_eq!(
wrong_kek.to_string(),
"backup bundle requires KEK 'backup-kek-1' version 3; supplied KEK 'backup-kek-2' version 1 cannot open it"
);
let unknown = BackupError::UnknownVersion { found: 9, supported: 1 };
assert_eq!(
unknown.to_string(),
"unknown backup manifest format version 9 (this build supports version 1)"
);
assert_eq!(
BackupError::missing_artifact("key-material").to_string(),
"backup bundle is missing a required artifact: key-material"
);
assert_eq!(
BackupError::incomplete_bundle("manifest has no completeness marker").to_string(),
"backup bundle is incomplete (manifest has no completeness marker); incomplete bundles must never be restored"
);
}
}
-999
View File
@@ -1,999 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Local backend backup export: sealed, KEK-protected bundle production.
//!
//! This is the producer side only; restore lives in a follow-up change. The
//! admin API is not wired here either — callers construct the request and
//! supply the backup KEK explicitly.
//!
//! # Bundle layout
//!
//! A bundle is a directory (simple to produce, artifacts stream one file at a
//! time, and partial output is trivially recognizable because the manifest is
//! written last):
//!
//! ```text
//! <destination>/
//! manifest.json # sealed BackupManifest, written last
//! artifacts/keys/<key_id>.key.enc # one per stored key record
//! artifacts/master-key.salt.enc # present when the salt file exists
//! ```
//!
//! # Artifact payload framing
//!
//! Every artifact payload is `nonce (12 bytes) || AES-256-GCM ciphertext`,
//! encrypted under the caller-supplied backup KEK with an AAD binding of
//! `(context, backup_id, snapshot_generation, artifact path)`, so an artifact
//! cannot be swapped into another bundle or renamed within its own bundle.
//! Records already encrypted at rest stay encrypted inside the wrap;
//! plaintext-dev-only records become ciphertext-only in the bundle, which is
//! their mandatory re-wrap under the backup KEK.
//!
//! # Write protocol
//!
//! Artifacts are written and fsynced first, re-read and digest-verified, and
//! only then is the sealed manifest (completeness marker plus final digest)
//! published. A crash at any earlier point leaves a bundle without a
//! manifest, which decodes as an incomplete bundle and can never be restored.
use crate::backends::local::{LocalKmsClient, StoredKeyProtection};
use crate::backup::capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
use crate::backup::error::BackupError;
use crate::backup::manifest::{
AeadAlgorithm, ArtifactDescriptor, ArtifactKind, BackupKekDescriptor, BackupManifest, CompletenessState, ContentDigest,
DigestAlgorithm, LocalKdfDescriptor, LocalKeyDerivation,
};
use crate::error::{KmsError, Result};
use aes_gcm::{
Aes256Gcm, Key, Nonce,
aead::{Aead, KeyInit, Payload},
};
use jiff::Zoned;
use rand::RngExt;
use serde::Deserialize;
use std::path::{Path, PathBuf};
use tokio::fs;
use tokio::io::AsyncWriteExt;
use zeroize::Zeroizing;
/// File name of the sealed manifest inside a bundle directory.
pub const LOCAL_BUNDLE_MANIFEST_FILE: &str = "manifest.json";
const ARTIFACTS_DIR: &str = "artifacts";
const KEYS_DIR: &str = "artifacts/keys";
const SALT_ARTIFACT_PATH: &str = "artifacts/master-key.salt.enc";
const AEAD_NONCE_LEN: usize = 12;
/// Domain-separation context for the artifact AAD binding.
const BUNDLE_AAD_CONTEXT: &str = "rustfs-kms-local-backup:v1";
/// Caller-supplied backup KEK: a trust root deliberately separate from the
/// business KMS hierarchy (it must not be a key that is itself part of the
/// state being backed up). Where the KEK comes from is the admin layer's
/// concern; this module only consumes it.
pub struct BackupKek {
kek_id: String,
kek_version: u32,
key: Zeroizing<[u8; 32]>,
}
impl BackupKek {
/// Wrap 32 bytes of KEK material. The material is zeroized on drop;
/// callers should zeroize their own copy of the input.
pub fn new(kek_id: impl Into<String>, kek_version: u32, key: [u8; 32]) -> Result<Self> {
let kek_id = kek_id.into();
if kek_id.is_empty() {
return Err(KmsError::validation_error("backup KEK id must not be empty"));
}
Ok(Self {
kek_id,
kek_version,
key: Zeroizing::new(key),
})
}
/// Manifest descriptor for this KEK.
pub fn descriptor(&self) -> BackupKekDescriptor {
BackupKekDescriptor {
kek_id: self.kek_id.clone(),
kek_version: self.kek_version,
aead_algorithm: AeadAlgorithm::Aes256Gcm,
}
}
fn cipher(&self) -> Aes256Gcm {
Aes256Gcm::new(&Key::<Aes256Gcm>::from(*self.key))
}
}
/// Parameters of one export run.
///
/// `snapshot_generation` is injected by the caller: the contract only
/// requires it to be monotonic per deployment, and the source (persisted
/// counter, coordinated clock) is decided by the admin layer, which keeps
/// this module free of ambient time or state lookups.
#[derive(Debug, Clone)]
pub struct LocalBackupExportRequest {
/// Unique identifier for this backup.
pub backup_id: String,
/// Opaque identity of the producing deployment.
pub deployment_identity: String,
/// RustFS version string recorded in the manifest.
pub rustfs_version: String,
/// Monotonic snapshot generation this bundle belongs to.
pub snapshot_generation: u64,
/// Bundle output directory; must not exist yet or must be empty.
pub destination: PathBuf,
}
impl LocalBackupExportRequest {
fn validate(&self) -> Result<()> {
for (field, value) in [
("backup_id", &self.backup_id),
("deployment_identity", &self.deployment_identity),
("rustfs_version", &self.rustfs_version),
] {
if value.is_empty() {
return Err(KmsError::validation_error(format!("backup export {field} must not be empty")));
}
}
Ok(())
}
}
/// Minimal projection of a stored key record: only the fields the exporter
/// needs. Unknown fields are ignored on purpose — the record travels into the
/// bundle byte-identical, so the exporter must not constrain its schema.
#[derive(Deserialize)]
struct StoredRecordProbe {
key_id: String,
#[serde(default)]
at_rest_protection: StoredKeyProtection,
}
struct CollectedRecord {
key_id: String,
protection: StoredKeyProtection,
/// Raw record bytes exactly as stored. Zeroized on drop because
/// plaintext-dev-only records embed key material.
raw: Zeroizing<Vec<u8>>,
}
struct CollectedSnapshot {
records: Vec<CollectedRecord>,
salt: Option<Vec<u8>>,
}
/// Export the Local backend's key directory as a sealed backup bundle.
///
/// The directory scan runs under the export fence, so concurrent
/// create/update/delete operations are either fully included or fully
/// excluded — never half a record. Encryption and bundle writing happen after
/// the fence is released to keep it short.
///
/// Returns the sealed manifest that was written to the bundle.
pub async fn export_local_backup(
client: &LocalKmsClient,
kek: &BackupKek,
request: &LocalBackupExportRequest,
) -> Result<BackupManifest> {
request.validate()?;
prepare_destination(&request.destination).await?;
let snapshot = collect_snapshot(client).await?;
if snapshot.records.is_empty() {
return Err(KmsError::invalid_operation(
"Local backup export found no key records; refusing to publish an empty bundle",
));
}
let has_encrypted = snapshot
.records
.iter()
.any(|record| record.protection == StoredKeyProtection::EncryptedMasterKey);
if has_encrypted && snapshot.salt.is_none() {
return Err(KmsError::invalid_operation(
"key directory contains encrypted-master-key records but the master key salt file is missing; \
the bundle would be unrestorable",
));
}
let manifest = build_and_write_bundle(kek, request, &snapshot).await?;
Ok(manifest)
}
/// Read and fully validate the manifest of a local bundle directory.
///
/// A directory without a manifest is an interrupted export: the manifest is
/// written last, so its absence means the bundle never sealed.
pub async fn read_local_bundle_manifest(bundle_dir: &Path) -> Result<BackupManifest> {
let manifest_path = bundle_dir.join(LOCAL_BUNDLE_MANIFEST_FILE);
let bytes = match fs::read(&manifest_path).await {
Ok(bytes) => bytes,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
return Err(BackupError::incomplete_bundle("bundle has no manifest; the export never sealed it").into());
}
Err(error) => return Err(error.into()),
};
let manifest = BackupManifest::decode(&bytes)?;
if manifest.backend != BackupBackendKind::Local {
return Err(
BackupError::corrupted(format!("bundle manifest declares backend {:?}, expected Local", manifest.backend)).into(),
);
}
Ok(manifest)
}
/// Read, verify, and decrypt one artifact of a local bundle.
///
/// Fail-closed order: KEK identity, artifact presence, declared length,
/// encrypted digest, then AEAD authentication. The returned plaintext is
/// zeroized on drop.
pub async fn decrypt_bundle_artifact(
bundle_dir: &Path,
manifest: &BackupManifest,
descriptor: &ArtifactDescriptor,
kek: &BackupKek,
) -> Result<Zeroizing<Vec<u8>>> {
manifest.backup_kek.ensure_matches(&kek.kek_id, kek.kek_version)?;
if descriptor.aead_algorithm != AeadAlgorithm::Aes256Gcm {
return Err(KmsError::unsupported_algorithm(format!(
"{:?} (local bundles are produced with AES-256-GCM)",
descriptor.aead_algorithm
)));
}
let artifact_path = bundle_dir.join(&descriptor.path);
let payload = match fs::read(&artifact_path).await {
Ok(payload) => payload,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
return Err(BackupError::missing_artifact(descriptor.path.clone()).into());
}
Err(error) => return Err(error.into()),
};
if (payload.len() as u64) < descriptor.len {
return Err(BackupError::truncated(format!(
"artifact '{}' is {} bytes, manifest declares {}",
descriptor.path,
payload.len(),
descriptor.len
))
.into());
}
if payload.len() as u64 != descriptor.len {
return Err(BackupError::corrupted(format!(
"artifact '{}' is {} bytes, manifest declares {}",
descriptor.path,
payload.len(),
descriptor.len
))
.into());
}
if ContentDigest::sha256_of(&payload) != descriptor.encrypted_digest {
return Err(BackupError::corrupted(format!("artifact '{}' does not match its manifest digest", descriptor.path)).into());
}
if payload.len() < AEAD_NONCE_LEN {
return Err(BackupError::corrupted(format!("artifact '{}' is too short to carry a nonce", descriptor.path)).into());
}
let (nonce_bytes, ciphertext) = payload.split_at(AEAD_NONCE_LEN);
let mut nonce = [0u8; AEAD_NONCE_LEN];
nonce.copy_from_slice(nonce_bytes);
let aad = artifact_aad(&manifest.backup_id, manifest.snapshot_generation, &descriptor.path);
let plaintext = kek
.cipher()
.decrypt(
&Nonce::from(nonce),
Payload {
msg: ciphertext,
aad: &aad,
},
)
.map_err(|_| {
KmsError::from(BackupError::corrupted(format!(
"artifact '{}' failed authenticated decryption under the supplied backup KEK",
descriptor.path
)))
})?;
Ok(Zeroizing::new(plaintext))
}
/// Scan the key directory under the export fence.
async fn collect_snapshot(client: &LocalKmsClient) -> Result<CollectedSnapshot> {
let _fence = client.acquire_export_fence().await;
let mut records = Vec::new();
let mut entries = fs::read_dir(client.key_directory()).await?;
while let Some(entry) = entries.next_entry().await? {
let path = entry.path();
if !path.extension().is_some_and(|extension| extension == "key") {
continue;
}
let stem = path
.file_stem()
.and_then(|stem| stem.to_str())
.ok_or_else(|| KmsError::configuration_error("Local KMS key file name must be valid UTF-8"))?
.to_string();
let raw = Zeroizing::new(fs::read(&path).await?);
// Any unreadable record aborts the export: a bundle silently missing
// one key is worse than no bundle at all.
let probe: StoredRecordProbe = serde_json::from_slice(&raw)
.map_err(|error| KmsError::material_corrupt(&stem, format!("stored key record does not deserialize: {error}")))?;
if probe.key_id != stem {
return Err(KmsError::invalid_key(format!(
"Local KMS key file identity mismatch: expected {stem:?}, found {:?}",
probe.key_id
)));
}
records.push(CollectedRecord {
key_id: stem,
protection: probe.at_rest_protection,
raw,
});
}
let salt_path = client.master_key_salt_file();
let salt = match fs::read(&salt_path).await {
Ok(bytes) => Some(bytes),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => None,
Err(error) => return Err(error.into()),
};
records.sort_by(|a, b| a.key_id.cmp(&b.key_id));
Ok(CollectedSnapshot { records, salt })
}
async fn build_and_write_bundle(
kek: &BackupKek,
request: &LocalBackupExportRequest,
snapshot: &CollectedSnapshot,
) -> Result<BackupManifest> {
let mut artifacts = Vec::with_capacity(snapshot.records.len() + 1);
for record in &snapshot.records {
let artifact_path = format!("{KEYS_DIR}/{}.key.enc", record.key_id);
let descriptor = encrypt_and_write_artifact(kek, request, ArtifactKind::KeyMaterial, &artifact_path, &record.raw).await?;
artifacts.push(descriptor);
}
if let Some(salt) = &snapshot.salt {
let descriptor = encrypt_and_write_artifact(kek, request, ArtifactKind::MasterKeySalt, SALT_ARTIFACT_PATH, salt).await?;
artifacts.push(descriptor);
}
// Make the artifact directory entries durable before sealing: the sealed
// manifest must never survive a crash that its artifacts did not.
fsync_dir(&request.destination.join(KEYS_DIR)).await?;
fsync_dir(&request.destination.join(ARTIFACTS_DIR)).await?;
let manifest = BackupManifest {
format_version: BackupManifest::FORMAT_VERSION,
backup_id: request.backup_id.clone(),
// Normalized to UTC so the stored spelling is host-independent: the
// local zone's name (or a POSIX TZ string in minimal containers) has
// no business inside a portable bundle.
created_at: Zoned::now().with_time_zone(jiff::tz::TimeZone::UTC),
rustfs_version: request.rustfs_version.clone(),
deployment_identity: request.deployment_identity.clone(),
backend: BackupBackendKind::Local,
at_rest_protection: weakest_observed_protection(&snapshot.records),
responsibility: BackupResponsibility::FullMaterial,
snapshot_generation: request.snapshot_generation,
backup_kek: kek.descriptor(),
artifacts,
local_kdf: Some(local_kdf_descriptor(snapshot)),
key_versions: None,
capability_discovery: None,
completeness: CompletenessState::InProgress,
manifest_digest: ContentDigest {
algorithm: DigestAlgorithm::Sha256,
hex: String::new(),
},
};
let manifest = manifest.seal()?;
let manifest_bytes = manifest.encode()?;
write_new_file(&request.destination.join(LOCAL_BUNDLE_MANIFEST_FILE), &manifest_bytes).await?;
fsync_dir(&request.destination).await?;
Ok(manifest)
}
/// Encrypt one artifact, write it durably, and re-read it to verify the
/// digest before it is allowed into the manifest.
async fn encrypt_and_write_artifact(
kek: &BackupKek,
request: &LocalBackupExportRequest,
kind: ArtifactKind,
artifact_path: &str,
plaintext: &[u8],
) -> Result<ArtifactDescriptor> {
let mut nonce = [0u8; AEAD_NONCE_LEN];
rand::rng().fill(&mut nonce[..]);
let aad = artifact_aad(&request.backup_id, request.snapshot_generation, artifact_path);
let ciphertext = kek
.cipher()
.encrypt(
&Nonce::from(nonce),
Payload {
msg: plaintext,
aad: &aad,
},
)
.map_err(|error| KmsError::cryptographic_error("backup_artifact_encrypt", error.to_string()))?;
let mut payload = Vec::with_capacity(AEAD_NONCE_LEN + ciphertext.len());
payload.extend_from_slice(&nonce);
payload.extend_from_slice(&ciphertext);
let absolute_path = request.destination.join(artifact_path);
write_new_file(&absolute_path, &payload).await?;
// Verify what actually landed on disk, not the in-memory buffer.
let written = fs::read(&absolute_path).await?;
let digest = ContentDigest::sha256_of(&written);
if written != payload {
return Err(KmsError::internal_error(format!(
"bundle artifact '{artifact_path}' read back differently than written"
)));
}
Ok(ArtifactDescriptor {
kind,
path: artifact_path.to_string(),
len: payload.len() as u64,
aead_algorithm: AeadAlgorithm::Aes256Gcm,
encrypted_digest: digest,
})
}
/// AAD binding an artifact to its bundle identity and path. A JSON tuple
/// gives unambiguous field boundaries without a hand-rolled framing format.
fn artifact_aad(backup_id: &str, snapshot_generation: u64, artifact_path: &str) -> Vec<u8> {
serde_json::to_vec(&(BUNDLE_AAD_CONTEXT, backup_id, snapshot_generation, artifact_path))
.expect("AAD tuple of strings and integers always serializes")
}
/// The bundle-level protection label is the weakest state observed across
/// records: any plaintext-dev-only record marks the whole bundle, then any
/// legacy-unspecified marker (unknown until read), and only a uniformly
/// encrypted directory is labeled encrypted-master-key.
fn weakest_observed_protection(records: &[CollectedRecord]) -> AtRestProtection {
let mut has_legacy = false;
for record in records {
match record.protection {
StoredKeyProtection::PlaintextDevOnly => return AtRestProtection::PlaintextDevOnly,
StoredKeyProtection::LegacyUnspecified => has_legacy = true,
StoredKeyProtection::EncryptedMasterKey => {}
}
}
if has_legacy {
AtRestProtection::LegacyUnspecified
} else {
AtRestProtection::EncryptedMasterKey
}
}
fn local_kdf_descriptor(snapshot: &CollectedSnapshot) -> LocalKdfDescriptor {
let mut modes = Vec::new();
for (marker, mode) in [
(StoredKeyProtection::EncryptedMasterKey, AtRestProtection::EncryptedMasterKey),
(StoredKeyProtection::PlaintextDevOnly, AtRestProtection::PlaintextDevOnly),
(StoredKeyProtection::LegacyUnspecified, AtRestProtection::LegacyUnspecified),
] {
if snapshot.records.iter().any(|record| record.protection == marker) {
modes.push(mode);
}
}
// With a salt on disk the backend derives via Argon2id; without one only
// the pre-beta.9 SHA-256 derivation can apply. For plaintext-only
// directories the derivation is informational.
let derivation = if snapshot.salt.is_some() {
LocalKeyDerivation::current_argon2id()
} else {
LocalKeyDerivation::LegacySha256
};
LocalKdfDescriptor {
derivation,
protection_modes: modes,
// The verifier shape is left to the restore change; the schema keeps
// it optional so bundles without one stay valid.
master_key_verifier: None,
}
}
async fn prepare_destination(destination: &Path) -> Result<()> {
if fs::try_exists(destination).await? {
let mut entries = fs::read_dir(destination)
.await
.map_err(|error| KmsError::invalid_operation(format!("backup destination is not a readable directory: {error}")))?;
if entries.next_entry().await?.is_some() {
return Err(KmsError::invalid_operation(
"backup destination directory is not empty; refusing to mix bundles",
));
}
}
fs::create_dir_all(destination.join(KEYS_DIR)).await?;
Ok(())
}
async fn write_new_file(path: &Path, bytes: &[u8]) -> Result<()> {
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(path)
.await
.map_err(|error| KmsError::io_error(format!("failed to create bundle file {}: {error}", path.display())))?;
file.write_all(bytes).await?;
file.sync_all().await?;
Ok(())
}
/// Fsync a directory so freshly created bundle entries survive power loss.
/// No-op on non-Unix platforms where directories cannot be opened for
/// syncing (mirrors the local backend's durable commit helper).
async fn fsync_dir(path: &Path) -> Result<()> {
#[cfg(unix)]
{
let path = path.to_path_buf();
tokio::task::spawn_blocking(move || std::fs::File::open(&path)?.sync_all())
.await
.map_err(|error| KmsError::io_error(error.to_string()))??;
}
#[cfg(not(unix))]
let _ = path;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::backends::KmsClient;
use crate::config::LocalConfig;
use std::sync::Arc;
use tempfile::TempDir;
async fn encrypted_client() -> (LocalKmsClient, TempDir) {
let temp = TempDir::new().expect("temp dir");
let client = LocalKmsClient::new(LocalConfig {
key_dir: temp.path().to_path_buf(),
master_key: Some("test-master-key".to_string()),
file_permissions: Some(0o600),
})
.await
.expect("client should initialize");
(client, temp)
}
async fn dev_client() -> (LocalKmsClient, TempDir) {
let temp = TempDir::new().expect("temp dir");
let client = LocalKmsClient::new(LocalConfig {
key_dir: temp.path().to_path_buf(),
master_key: None,
file_permissions: Some(0o600),
})
.await
.expect("client should initialize");
(client, temp)
}
fn test_kek() -> BackupKek {
BackupKek::new("backup-kek-test", 1, [0x42; 32]).expect("kek")
}
fn export_request(destination: PathBuf) -> LocalBackupExportRequest {
LocalBackupExportRequest {
backup_id: "backup-0001".to_string(),
deployment_identity: "deployment-test".to_string(),
rustfs_version: "1.0.0-test".to_string(),
snapshot_generation: 7,
destination,
}
}
fn walk_files(dir: &Path, out: &mut Vec<PathBuf>) {
for entry in std::fs::read_dir(dir).expect("read dir") {
let path = entry.expect("dir entry").path();
if path.is_dir() {
walk_files(&path, out);
} else {
out.push(path);
}
}
}
fn contains_subslice(haystack: &[u8], needle: &[u8]) -> bool {
!needle.is_empty() && haystack.windows(needle.len()).any(|window| window == needle)
}
#[tokio::test]
async fn export_round_trips_and_decrypts_to_source_records() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("alpha", "AES_256", None).await.expect("create alpha");
client.create_key("beta", "AES_256", None).await.expect("create beta");
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed");
assert_eq!(manifest.backend, BackupBackendKind::Local);
assert_eq!(manifest.responsibility, BackupResponsibility::FullMaterial);
assert_eq!(manifest.at_rest_protection, AtRestProtection::EncryptedMasterKey);
assert_eq!(manifest.snapshot_generation, 7);
let kdf = manifest.local_kdf.as_ref().expect("local kdf descriptor");
assert_eq!(kdf.derivation, LocalKeyDerivation::current_argon2id());
assert_eq!(kdf.protection_modes, vec![AtRestProtection::EncryptedMasterKey]);
// alpha, beta (sorted), then the salt artifact.
assert_eq!(manifest.artifacts.len(), 3);
assert_eq!(manifest.artifacts[0].path, "artifacts/keys/alpha.key.enc");
assert_eq!(manifest.artifacts[1].path, "artifacts/keys/beta.key.enc");
assert_eq!(manifest.artifacts[2].kind, ArtifactKind::MasterKeySalt);
let reread = read_local_bundle_manifest(&destination)
.await
.expect("manifest should decode");
assert_eq!(reread, manifest);
for (artifact, key_id) in [(&manifest.artifacts[0], "alpha"), (&manifest.artifacts[1], "beta")] {
let decrypted = decrypt_bundle_artifact(&destination, &manifest, artifact, &kek)
.await
.expect("artifact should decrypt");
let source = fs::read(client.key_directory().join(format!("{key_id}.key")))
.await
.expect("source record");
assert_eq!(decrypted.as_slice(), source.as_slice(), "record {key_id} must round-trip verbatim");
}
let salt = decrypt_bundle_artifact(&destination, &manifest, &manifest.artifacts[2], &kek)
.await
.expect("salt should decrypt");
let source_salt = fs::read(client.master_key_salt_file()).await.expect("source salt");
assert_eq!(salt.as_slice(), source_salt.as_slice());
}
#[tokio::test]
async fn plaintext_dev_only_material_is_rewrapped_and_absent_from_bundle() {
let (client, _key_dir) = dev_client().await;
client.create_key("dev-key", "AES_256", None).await.expect("create key");
let material = client
.decrypt_key_material_for_export("dev-key")
.await
.expect("material should be readable");
let source_record = fs::read(client.key_directory().join("dev-key.key")).await.expect("record");
let record_json: serde_json::Value = serde_json::from_slice(&source_record).expect("record parses");
let material_base64 = record_json
.get("encrypted_key_material")
.and_then(|value| value.as_str())
.expect("material field")
.to_string();
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed");
assert_eq!(manifest.at_rest_protection, AtRestProtection::PlaintextDevOnly);
let kdf = manifest.local_kdf.as_ref().expect("local kdf descriptor");
assert_eq!(kdf.protection_modes, vec![AtRestProtection::PlaintextDevOnly]);
assert_eq!(kdf.derivation, LocalKeyDerivation::LegacySha256);
assert!(
!manifest.artifacts.iter().any(|a| a.kind == ArtifactKind::MasterKeySalt),
"dev-mode directory has no salt to bundle"
);
// Byte-level: neither the raw material nor its base64 form may appear
// anywhere in the bundle. The mandatory KEK re-wrap is what hides it.
let mut files = Vec::new();
walk_files(&destination, &mut files);
assert!(!files.is_empty());
for file in files {
let bytes = std::fs::read(&file).expect("bundle file");
assert!(
!contains_subslice(&bytes, material.as_ref()),
"raw key material leaked into {}",
file.display()
);
assert!(
!contains_subslice(&bytes, material_base64.as_bytes()),
"base64 key material leaked into {}",
file.display()
);
}
// The wrapped record still round-trips for restore.
let decrypted = decrypt_bundle_artifact(&destination, &manifest, &manifest.artifacts[0], &kek)
.await
.expect("artifact should decrypt");
assert_eq!(decrypted.as_slice(), source_record.as_slice());
}
#[tokio::test]
async fn export_fence_blocks_writers_until_released() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("existing", "AES_256", None).await.expect("create key");
let client = Arc::new(client);
let fence = client.acquire_export_fence().await;
let writer = {
let client = Arc::clone(&client);
tokio::spawn(async move {
client.create_key("new-key", "AES_256", None).await.expect("create");
client.disable_key("existing", None).await.expect("disable");
})
};
for _ in 0..64 {
tokio::task::yield_now().await;
}
assert!(!writer.is_finished(), "writers must stay blocked while the export fence is held");
drop(fence);
writer.await.expect("writer should finish after fence release");
assert!(
fs::try_exists(client.key_directory().join("new-key.key"))
.await
.expect("exists")
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_writers_yield_complete_records() {
let (client, _key_dir) = encrypted_client().await;
for index in 0..5 {
client
.create_key(&format!("seed-{index}"), "AES_256", None)
.await
.expect("seed key");
}
let client = Arc::new(client);
let writer = {
let client = Arc::clone(&client);
tokio::spawn(async move {
for index in 0..30 {
client
.create_key(&format!("concurrent-{index}"), "AES_256", None)
.await
.expect("create");
let target = format!("seed-{}", index % 5);
if index % 2 == 0 {
client.disable_key(&target, None).await.expect("disable");
} else {
client.enable_key(&target, None).await.expect("enable");
}
}
})
};
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed under concurrent writers");
writer.await.expect("writer task");
// Whatever subset of writers landed before the fence, every record in
// the bundle must be complete: parseable, self-identifying, and with
// non-empty material. No torn records, no half-updates.
let reread = read_local_bundle_manifest(&destination).await.expect("manifest decodes");
assert_eq!(reread, manifest);
for artifact in manifest.artifacts.iter().filter(|a| a.kind == ArtifactKind::KeyMaterial) {
let record = decrypt_bundle_artifact(&destination, &manifest, artifact, &kek)
.await
.expect("record decrypts");
let value: serde_json::Value = serde_json::from_slice(&record).expect("record is complete JSON");
let key_id = value.get("key_id").and_then(|v| v.as_str()).expect("key_id present");
assert_eq!(artifact.path, format!("artifacts/keys/{key_id}.key.enc"));
let material = value
.get("encrypted_key_material")
.and_then(|v| v.as_str())
.expect("material present");
assert!(!material.is_empty());
}
}
#[tokio::test]
async fn tampered_and_truncated_bundles_fail_closed() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("victim", "AES_256", None).await.expect("create key");
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed");
let artifact = &manifest.artifacts[0];
let artifact_file = destination.join(&artifact.path);
let original_artifact = std::fs::read(&artifact_file).expect("artifact bytes");
// Tampered artifact byte: digest verification rejects it.
let mut tampered = original_artifact.clone();
let last = tampered.len() - 1;
tampered[last] ^= 0x01;
std::fs::write(&artifact_file, &tampered).expect("write tampered");
let error = decrypt_bundle_artifact(&destination, &manifest, artifact, &kek)
.await
.expect_err("tampered artifact must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::Corrupted { .. })), "got {error:?}");
// Truncated artifact: typed truncation error.
std::fs::write(&artifact_file, &original_artifact[..original_artifact.len() - 4]).expect("truncate");
let error = decrypt_bundle_artifact(&destination, &manifest, artifact, &kek)
.await
.expect_err("truncated artifact must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::Truncated { .. })), "got {error:?}");
std::fs::write(&artifact_file, &original_artifact).expect("restore artifact");
// Tampered manifest (generation flip): sealed digest mismatch.
let manifest_file = destination.join(LOCAL_BUNDLE_MANIFEST_FILE);
let original_manifest = std::fs::read(&manifest_file).expect("manifest bytes");
let tampered_manifest = String::from_utf8(original_manifest.clone())
.expect("manifest is utf-8")
.replace("\"snapshot_generation\":7", "\"snapshot_generation\":8");
assert_ne!(tampered_manifest.as_bytes(), original_manifest.as_slice(), "tamper must apply");
std::fs::write(&manifest_file, tampered_manifest).expect("write tampered manifest");
let error = read_local_bundle_manifest(&destination)
.await
.expect_err("tampered manifest must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::Corrupted { .. })), "got {error:?}");
// Truncated manifest.
std::fs::write(&manifest_file, &original_manifest[..original_manifest.len() / 2]).expect("truncate manifest");
let error = read_local_bundle_manifest(&destination)
.await
.expect_err("truncated manifest must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::Truncated { .. })), "got {error:?}");
// Missing manifest: the bundle never sealed.
std::fs::remove_file(&manifest_file).expect("remove manifest");
let error = read_local_bundle_manifest(&destination)
.await
.expect_err("bundle without manifest must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::IncompleteBundle { .. })), "got {error:?}");
}
#[tokio::test]
async fn wrong_kek_is_rejected_before_decryption() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("victim", "AES_256", None).await.expect("create key");
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed");
let artifact = &manifest.artifacts[0];
let wrong_id = BackupKek::new("other-kek", 1, [0x42; 32]).expect("kek");
let error = decrypt_bundle_artifact(&destination, &manifest, artifact, &wrong_id)
.await
.expect_err("mismatched KEK id must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::WrongKek { .. })), "got {error:?}");
let wrong_version = BackupKek::new("backup-kek-test", 2, [0x42; 32]).expect("kek");
let error = decrypt_bundle_artifact(&destination, &manifest, artifact, &wrong_version)
.await
.expect_err("mismatched KEK version must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::WrongKek { .. })), "got {error:?}");
// Right identity, wrong material: AEAD authentication fails closed.
let wrong_material = BackupKek::new("backup-kek-test", 1, [0x24; 32]).expect("kek");
let error = decrypt_bundle_artifact(&destination, &manifest, artifact, &wrong_material)
.await
.expect_err("wrong KEK material must be rejected");
assert!(matches!(error, KmsError::Backup(BackupError::Corrupted { .. })), "got {error:?}");
}
#[tokio::test]
async fn missing_salt_with_encrypted_records_fails_export() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("victim", "AES_256", None).await.expect("create key");
fs::remove_file(client.master_key_salt_file()).await.expect("remove salt");
let bundle = TempDir::new().expect("bundle dir");
let error = export_local_backup(&client, &test_kek(), &export_request(bundle.path().join("bundle")))
.await
.expect_err("export without salt must fail");
assert!(matches!(error, KmsError::InvalidOperation { .. }), "got {error:?}");
assert!(error.to_string().contains("salt"), "got {error}");
}
#[tokio::test]
async fn refuses_empty_key_dir_and_nonempty_destination() {
let (client, _key_dir) = dev_client().await;
let bundle = TempDir::new().expect("bundle dir");
let error = export_local_backup(&client, &test_kek(), &export_request(bundle.path().join("bundle")))
.await
.expect_err("empty key dir must not produce a bundle");
assert!(matches!(error, KmsError::InvalidOperation { .. }), "got {error:?}");
client.create_key("dev-key", "AES_256", None).await.expect("create key");
let occupied = bundle.path().join("occupied");
std::fs::create_dir_all(&occupied).expect("mkdir");
std::fs::write(occupied.join("stale"), b"leftover").expect("occupy");
let error = export_local_backup(&client, &test_kek(), &export_request(occupied))
.await
.expect_err("non-empty destination must be refused");
assert!(matches!(error, KmsError::InvalidOperation { .. }), "got {error:?}");
}
#[tokio::test]
async fn legacy_records_export_verbatim_with_weakest_protection_label() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("modern", "AES_256", None).await.expect("create key");
client.create_key("legacy-key", "AES_256", None).await.expect("create key");
// Strip the protection marker to fabricate a pre-beta.9 record, the
// same way the local backend's own legacy-compat tests do.
let legacy_path = client.key_directory().join("legacy-key.key");
let mut record: serde_json::Value =
serde_json::from_slice(&fs::read(&legacy_path).await.expect("record")).expect("record parses");
record
.as_object_mut()
.expect("record is an object")
.remove("at_rest_protection");
let legacy_bytes = serde_json::to_vec_pretty(&record).expect("record serializes");
fs::write(&legacy_path, &legacy_bytes).await.expect("write legacy record");
let bundle = TempDir::new().expect("bundle dir");
let destination = bundle.path().join("bundle");
let kek = test_kek();
let manifest = export_local_backup(&client, &kek, &export_request(destination.clone()))
.await
.expect("export should succeed");
assert_eq!(manifest.at_rest_protection, AtRestProtection::LegacyUnspecified);
let kdf = manifest.local_kdf.as_ref().expect("local kdf descriptor");
assert_eq!(
kdf.protection_modes,
vec![AtRestProtection::EncryptedMasterKey, AtRestProtection::LegacyUnspecified]
);
let legacy_artifact = manifest
.artifacts
.iter()
.find(|a| a.path == "artifacts/keys/legacy-key.key.enc")
.expect("legacy artifact");
let decrypted = decrypt_bundle_artifact(&destination, &manifest, legacy_artifact, &kek)
.await
.expect("legacy artifact decrypts");
assert_eq!(decrypted.as_slice(), legacy_bytes.as_slice(), "legacy record must travel verbatim");
}
#[tokio::test]
async fn record_identity_mismatch_aborts_export() {
let (client, _key_dir) = encrypted_client().await;
client.create_key("good", "AES_256", None).await.expect("create key");
std::fs::copy(client.key_directory().join("good.key"), client.key_directory().join("evil.key"))
.expect("plant mismatched record");
let bundle = TempDir::new().expect("bundle dir");
let error = export_local_backup(&client, &test_kek(), &export_request(bundle.path().join("bundle")))
.await
.expect_err("identity mismatch must abort the export");
assert!(matches!(error, KmsError::InvalidKey { .. }), "got {error:?}");
}
}
File diff suppressed because it is too large Load Diff
-68
View File
@@ -1,68 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Backup/restore contracts and backup production for KMS state.
//!
//! The contract side defines the versioned backup manifest, the per-backend
//! responsibility matrix, typed failure modes, and the restore dry-run
//! report. [`local_export`] implements the producer side for the Local
//! backend as a crate-internal API; restore orchestration and the admin API
//! build on these pieces in follow-up changes.
//!
//! # Bundle model
//!
//! A backup bundle is a set of AEAD-encrypted artifacts described by a single
//! [`BackupManifest`]. All state in a bundle belongs to one snapshot
//! generation — there is no partially consistent bundle. The bundle is
//! protected by a backup KEK that is deliberately outside the business KMS
//! trust hierarchy, and the manifest is sealed with a completeness marker and
//! a final digest; a bundle that never reached its marker is permanently
//! non-restorable.
//!
//! # Restore ordering
//!
//! Restore implementations must follow this order: re-establish the external
//! trust root first (Vault/HSM native restore where one exists), then
//! material and version records into staging, then metadata and
//! configuration, then verification, and only then an explicit atomic
//! cutover. A dry-run ([`RestoreDryRunReport`]) performs zero writes.
//!
//! # Deliberately unfrozen
//!
//! Fields whose shape depends on contracts still in flight are reserved
//! rather than guessed (see [`ReservedSlot`]): the per-key version inventory
//! (backlog#1565) and capability discovery (backlog#1571). Alias and policy
//! artifacts are reserved names for features that do not exist yet. Reserved
//! slots reject data in format version 1 and become real types in a later
//! format version.
mod capability;
mod dry_run;
mod error;
pub mod local_export;
mod manifest;
pub use capability::{AtRestProtection, BackupBackendKind, BackupResponsibility};
pub use dry_run::{
ExternalDependencyMismatch, RestoreBlocker, RestoreBlockerCode, RestoreConflict, RestoreConflictKind, RestoreDryRunReport,
};
pub use error::BackupError;
pub use local_export::{
BackupKek, LOCAL_BUNDLE_MANIFEST_FILE, LocalBackupExportRequest, decrypt_bundle_artifact, export_local_backup,
read_local_bundle_manifest,
};
pub use manifest::{
AeadAlgorithm, ArtifactDescriptor, ArtifactKind, BackupKekDescriptor, BackupManifest, CompletenessState, ContentDigest,
DigestAlgorithm, LocalKdfDescriptor, LocalKeyDerivation, ReservedSlot,
};
+20 -577
View File
@@ -29,23 +29,8 @@ pub const ENV_KMS_VAULT_TRANSIT_METADATA_KV_MOUNT: &str = "RUSTFS_KMS_VAULT_TRAN
pub const ENV_KMS_VAULT_TRANSIT_METADATA_PREFIX: &str = "RUSTFS_KMS_VAULT_TRANSIT_METADATA_PREFIX";
pub const ENV_KMS_STATIC_SECRET_KEY: &str = "RUSTFS_KMS_STATIC_SECRET_KEY";
pub const ENV_KMS_STATIC_SECRET_KEY_FILE: &str = "RUSTFS_KMS_STATIC_SECRET_KEY_FILE";
pub const ENV_KMS_VAULT_APPROLE_ROLE_ID: &str = "RUSTFS_KMS_VAULT_APPROLE_ROLE_ID";
pub const ENV_KMS_VAULT_APPROLE_SECRET_ID: &str = "RUSTFS_KMS_VAULT_APPROLE_SECRET_ID";
pub const ENV_KMS_VAULT_APPROLE_SECRET_ID_FILE: &str = "RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE";
pub const ENV_KMS_VAULT_APPROLE_MOUNT: &str = "RUSTFS_KMS_VAULT_APPROLE_MOUNT";
pub const ENV_KMS_VAULT_TOKEN_FILE: &str = "RUSTFS_KMS_VAULT_TOKEN_FILE";
pub const DEFAULT_VAULT_TRANSIT_METADATA_KV_MOUNT: &str = "secret";
pub const DEFAULT_VAULT_TRANSIT_METADATA_KEY_PREFIX: &str = "rustfs/kms/transit-metadata";
pub const DEFAULT_VAULT_APPROLE_MOUNT: &str = "approle";
/// Upper bound applied to `KmsConfig::timeout` when deriving backend behavior.
///
/// Out-of-range values are clamped at use rather than rejected so existing
/// deployments with oversized settings keep starting after an upgrade.
pub(crate) const MAX_OPERATION_TIMEOUT: Duration = Duration::from_secs(300);
/// Upper bound applied to `KmsConfig::retry_attempts` when deriving backend behavior.
pub(crate) const MAX_RETRY_ATTEMPTS: u32 = 10;
fn default_vault_transit_metadata_kv_mount() -> String {
DEFAULT_VAULT_TRANSIT_METADATA_KV_MOUNT.to_string()
@@ -55,14 +40,6 @@ fn default_vault_transit_metadata_key_prefix() -> String {
DEFAULT_VAULT_TRANSIT_METADATA_KEY_PREFIX.to_string()
}
fn default_vault_kv2_mount_path() -> String {
"transit".to_string()
}
fn default_vault_approle_mount() -> String {
DEFAULT_VAULT_APPROLE_MOUNT.to_string()
}
pub const KMS_CONFIG_REDACTION_RULES: &[RedactionRule] = &[
RedactionRule::new("kms.local.master_key", RedactionLevel::Secret, "local backend key encryption material"),
RedactionRule::new("kms.vault.token", RedactionLevel::Secret, "vault authentication token"),
@@ -114,9 +91,7 @@ pub(crate) fn redacted_secret_option(value: Option<&str>) -> Option<&'static str
/// KMS backend types
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub enum KmsBackend {
/// Vault KV v2 storage backend: master key material is stored directly in KV v2.
/// Confidentiality relies on Vault ACLs, KV v2 at-rest encryption, and TLS; the
/// backend performs no Transit wrapping of key material.
/// Vault KV v2 + Transit backend (key metadata in KV, wrapping via Transit)
#[serde(rename = "VaultKV2", alias = "Vault")]
VaultKv2,
/// Vault Transit backend using Vault as the cryptographic source of truth
@@ -142,15 +117,9 @@ pub struct KmsConfig {
/// Allow development-only insecure defaults such as plaintext local keys or HTTP Vault.
#[serde(default)]
pub allow_insecure_dev_defaults: bool,
/// Timeout for a single backend attempt.
///
/// This bounds one outbound request, not the whole operation: the operation
/// policy owns the total deadline across retries. Values above 300 seconds
/// are clamped at use (see `KmsConfig::effective_timeout`).
/// Operation timeout
pub timeout: Duration,
/// Number of retry attempts.
///
/// Values above 10 are clamped at use (see `KmsConfig::effective_retry_attempts`).
/// Number of retry attempts
pub retry_attempts: u32,
/// Enable caching
pub enable_cache: bool,
@@ -178,7 +147,7 @@ impl Default for KmsConfig {
pub enum BackendConfig {
/// Local backend configuration
Local(LocalConfig),
/// Vault KV v2 storage backend configuration
/// Vault KV v2 + Transit backend configuration
#[serde(rename = "VaultKV2", alias = "Vault")]
VaultKv2(Box<VaultConfig>),
/// Vault Transit backend configuration
@@ -286,11 +255,7 @@ impl StaticConfig {
}
}
/// Vault KV v2 backend configuration.
///
/// Key material and metadata are stored directly in KV v2; any identity with KV read
/// access to the key path can recover plaintext master key material. Use the Vault
/// Transit backend when cryptographic isolation of key material is required.
/// Vault KV v2 + Transit backend configuration (metadata in KV, key wrapping via Transit)
#[derive(Clone, Serialize, Deserialize)]
pub struct VaultConfig {
/// Vault server URL
@@ -299,10 +264,7 @@ pub struct VaultConfig {
pub auth_method: VaultAuthMethod,
/// Vault namespace (Vault Enterprise)
pub namespace: Option<String>,
/// Deprecated: legacy Transit engine mount path. The Vault KV2 backend never calls
/// the Transit engine, so this value is unused; the field is retained (and
/// defaulted) only so previously persisted configurations keep deserializing.
#[serde(default = "default_vault_kv2_mount_path")]
/// Transit engine mount path
pub mount_path: String,
/// KV engine mount path for storing keys
pub kv_mount: String,
@@ -398,94 +360,18 @@ impl Default for VaultTransitConfig {
pub enum VaultAuthMethod {
/// Token authentication
Token { token: String },
/// AppRole authentication: login with `role_id` + `secret_id` for a
/// lease-bound token that is renewed in the background.
AppRole {
role_id: String,
/// Inline secret_id; used only when `secret_id_file` is unset.
secret_id: String,
/// Path to a file holding the secret_id. Re-read on every login so an
/// externally rotated secret_id is picked up; takes precedence over the
/// inline value.
#[serde(default)]
secret_id_file: Option<PathBuf>,
/// AppRole auth engine mount path.
#[serde(default = "default_vault_approle_mount")]
mount: String,
/// Fail-closed margin in seconds: once the current token is within this
/// window of expiry without a successful refresh, requests are refused
/// instead of sent with a token that may lapse mid-flight. Defaults to
/// the per-attempt timeout.
#[serde(default)]
refresh_safety_window_secs: Option<u64>,
},
/// Agent-managed token file (for example a Vault Agent auto-auth sink):
/// the token is read from `path` and re-read periodically so a token
/// rotated by the agent is picked up without a restart.
TokenFile {
path: PathBuf,
/// Seconds between token file re-reads. Each successful read also
/// extends the token's observed validity to twice this value, so a
/// file that stops being readable eventually trips the fail-closed
/// window. Defaults to 30 seconds.
#[serde(default)]
poll_interval_secs: Option<u64>,
/// Fail-closed margin in seconds, as on `AppRole`. Defaults to the
/// per-attempt timeout.
#[serde(default)]
refresh_safety_window_secs: Option<u64>,
},
}
impl VaultAuthMethod {
/// AppRole authentication with the default mount and no secret-id file.
pub fn approle(role_id: String, secret_id: String) -> Self {
Self::AppRole {
role_id,
secret_id,
secret_id_file: None,
mount: default_vault_approle_mount(),
refresh_safety_window_secs: None,
}
}
/// Agent-managed token file with the default poll interval.
pub fn token_file(path: PathBuf) -> Self {
Self::TokenFile {
path,
poll_interval_secs: None,
refresh_safety_window_secs: None,
}
}
/// AppRole authentication
AppRole { role_id: String, secret_id: String },
}
impl fmt::Debug for VaultAuthMethod {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Token { token } => f.debug_struct("Token").field("token", &redacted_secret(token)).finish(),
Self::AppRole {
role_id,
secret_id,
secret_id_file,
mount,
refresh_safety_window_secs,
} => f
Self::AppRole { role_id, secret_id } => f
.debug_struct("AppRole")
.field("role_id", role_id)
.field("secret_id", &redacted_secret(secret_id))
.field("secret_id_file", secret_id_file)
.field("mount", mount)
.field("refresh_safety_window_secs", refresh_safety_window_secs)
.finish(),
Self::TokenFile {
path,
poll_interval_secs,
refresh_safety_window_secs,
} => f
.debug_struct("TokenFile")
.field("path", path)
.field("poll_interval_secs", poll_interval_secs)
.field("refresh_safety_window_secs", refresh_safety_window_secs)
.finish(),
}
}
@@ -538,12 +424,7 @@ impl KmsConfig {
}
}
/// Create a new KMS configuration for the Vault KV v2 backend with token authentication.
///
/// Master key material is stored directly in Vault KV v2: confidentiality relies on
/// Vault ACLs, KV v2 at-rest encryption, and TLS. KV read access to the key path is
/// equivalent to holding the plaintext master keys. Use [`KmsConfig::vault_transit`]
/// when key material must never be readable through Vault storage APIs.
/// Create a new KMS configuration for Vault backend with token authentication (recommended for production)
pub fn vault(address: Url, token: String) -> Self {
Self {
backend: KmsBackend::VaultKv2,
@@ -556,16 +437,13 @@ impl KmsConfig {
}
}
/// Create a new KMS configuration for the Vault KV v2 backend with AppRole authentication.
///
/// Shares the security boundary described on [`KmsConfig::vault`]: key material lives
/// in KV v2 and is protected only by Vault ACLs and KV v2 at-rest encryption.
/// Create a new KMS configuration for Vault backend with AppRole authentication (recommended for production)
pub fn vault_approle(address: Url, role_id: String, secret_id: String) -> Self {
Self {
backend: KmsBackend::VaultKv2,
backend_config: BackendConfig::VaultKv2(Box::new(VaultConfig {
address: address.to_string(),
auth_method: VaultAuthMethod::approle(role_id, secret_id),
auth_method: VaultAuthMethod::AppRole { role_id, secret_id },
..Default::default()
})),
..Default::default()
@@ -658,16 +536,6 @@ impl KmsConfig {
self
}
/// Per-attempt timeout with the configured value clamped to the supported maximum.
pub(crate) fn effective_timeout(&self) -> Duration {
self.timeout.min(MAX_OPERATION_TIMEOUT)
}
/// Retry attempts with the configured value clamped to the supported maximum.
pub(crate) fn effective_retry_attempts(&self) -> u32 {
self.retry_attempts.min(MAX_RETRY_ATTEMPTS)
}
/// Validate the configuration
pub fn validate(&self) -> Result<()> {
// Validate timeout
@@ -680,23 +548,6 @@ impl KmsConfig {
return Err(KmsError::configuration_error("Retry attempts must be greater than 0"));
}
// Oversized values are clamped at use (not rejected) so pre-existing
// configurations cannot keep the service from starting after upgrade.
if self.timeout > MAX_OPERATION_TIMEOUT {
tracing::warn!(
configured_secs = self.timeout.as_secs(),
max_secs = MAX_OPERATION_TIMEOUT.as_secs(),
"KMS timeout exceeds the supported maximum; backend operations clamp it to the maximum"
);
}
if self.retry_attempts > MAX_RETRY_ATTEMPTS {
tracing::warn!(
configured = self.retry_attempts,
max = MAX_RETRY_ATTEMPTS,
"KMS retry_attempts exceeds the supported maximum; backend operations clamp it to the maximum"
);
}
// Validate backend-specific configuration
match &self.backend_config {
BackendConfig::Local(config) => {
@@ -723,14 +574,13 @@ impl KmsConfig {
return Err(KmsError::configuration_error("Vault KV2 address must use http or https scheme"));
}
validate_vault_auth_method("Vault KV2", &config.auth_method)?;
if !self.allow_insecure_dev_defaults {
validate_vault_development_defaults("Vault KV2", &config.address, &config.auth_method, config.tls.as_ref())?;
}
// `mount_path` is deprecated and unused by this backend, so an empty value
// is deliberately not an error.
if config.mount_path.is_empty() {
return Err(KmsError::configuration_error("Vault KV2 mount path cannot be empty"));
}
// Validate TLS configuration if using HTTPS
if config.address.starts_with("https://")
@@ -748,8 +598,6 @@ impl KmsConfig {
return Err(KmsError::configuration_error("Vault Transit address must use http or https scheme"));
}
validate_vault_auth_method("Vault Transit", &config.auth_method)?;
if !self.allow_insecure_dev_defaults {
validate_vault_development_defaults(
"Vault Transit",
@@ -854,24 +702,14 @@ impl KmsConfig {
}
KmsBackend::VaultKv2 => {
let address = get_env_str("RUSTFS_KMS_VAULT_ADDRESS", "http://localhost:8200");
let auth_method = vault_auth_method_from_env()?;
let token = get_env_str("RUSTFS_KMS_VAULT_TOKEN", "dev-token");
let skip_tls_verify = get_env_bool(ENV_KMS_VAULT_SKIP_TLS_VERIFY, false);
let mount_path = match get_env_opt_str("RUSTFS_KMS_VAULT_MOUNT_PATH") {
Some(path) => {
tracing::warn!(
"RUSTFS_KMS_VAULT_MOUNT_PATH is deprecated for the Vault KV2 backend: it never calls the Transit engine and the value is stored but unused"
);
path
}
None => default_vault_kv2_mount_path(),
};
config.backend_config = BackendConfig::VaultKv2(Box::new(VaultConfig {
address,
auth_method,
auth_method: VaultAuthMethod::Token { token },
namespace: get_env_opt_str("RUSTFS_KMS_VAULT_NAMESPACE"),
mount_path,
mount_path: get_env_str("RUSTFS_KMS_VAULT_MOUNT_PATH", "transit"),
kv_mount: get_env_str("RUSTFS_KMS_VAULT_KV_MOUNT", "secret"),
key_path_prefix: get_env_str("RUSTFS_KMS_VAULT_KEY_PREFIX", "rustfs/kms/keys"),
tls: vault_tls_config(skip_tls_verify),
@@ -879,12 +717,12 @@ impl KmsConfig {
}
KmsBackend::VaultTransit => {
let address = get_env_str("RUSTFS_KMS_VAULT_ADDRESS", "http://localhost:8200");
let auth_method = vault_auth_method_from_env()?;
let token = get_env_str("RUSTFS_KMS_VAULT_TOKEN", "dev-token");
let skip_tls_verify = get_env_bool(ENV_KMS_VAULT_SKIP_TLS_VERIFY, false);
config.backend_config = BackendConfig::VaultTransit(Box::new(VaultTransitConfig {
address,
auth_method,
auth_method: VaultAuthMethod::Token { token },
namespace: get_env_opt_str("RUSTFS_KMS_VAULT_NAMESPACE"),
mount_path: get_env_str("RUSTFS_KMS_VAULT_MOUNT_PATH", "transit"),
metadata_kv_mount: get_env_str(
@@ -963,95 +801,6 @@ fn is_under_temp_dir(path: &Path) -> bool {
path.starts_with(std::env::temp_dir())
}
/// Resolve the Vault auth method from environment variables.
///
/// Setting `RUSTFS_KMS_VAULT_APPROLE_ROLE_ID` selects AppRole authentication;
/// the secret_id then comes from `RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE`
/// (re-read on every login, mirroring the `RUSTFS_KMS_STATIC_SECRET_KEY_FILE`
/// precedent) or inline from `RUSTFS_KMS_VAULT_APPROLE_SECRET_ID`, with the
/// file taking precedence. Without a role id the legacy token flow applies.
fn vault_auth_method_from_env() -> Result<VaultAuthMethod> {
if let Some(token_file) = get_env_opt_str(ENV_KMS_VAULT_TOKEN_FILE) {
// A token file names one authoritative credential source; combining it
// with another one would leave the effective identity ambiguous, so
// that is a configuration error rather than a precedence rule.
if get_env_opt_str(ENV_KMS_VAULT_APPROLE_ROLE_ID).is_some() {
return Err(KmsError::configuration_error(format!(
"{ENV_KMS_VAULT_TOKEN_FILE} cannot be combined with {ENV_KMS_VAULT_APPROLE_ROLE_ID}; configure exactly one Vault auth method"
)));
}
if get_env_opt_str("RUSTFS_KMS_VAULT_TOKEN").is_some() {
return Err(KmsError::configuration_error(format!(
"{ENV_KMS_VAULT_TOKEN_FILE} cannot be combined with RUSTFS_KMS_VAULT_TOKEN; configure exactly one Vault auth method"
)));
}
return Ok(VaultAuthMethod::token_file(PathBuf::from(token_file)));
}
let Some(role_id) = get_env_opt_str(ENV_KMS_VAULT_APPROLE_ROLE_ID) else {
return Ok(VaultAuthMethod::Token {
token: get_env_str("RUSTFS_KMS_VAULT_TOKEN", "dev-token"),
});
};
let secret_id_file = get_env_opt_str(ENV_KMS_VAULT_APPROLE_SECRET_ID_FILE).map(PathBuf::from);
let secret_id = get_env_opt_str(ENV_KMS_VAULT_APPROLE_SECRET_ID).unwrap_or_default();
if secret_id.is_empty() && secret_id_file.is_none() {
return Err(KmsError::configuration_error(format!(
"Vault AppRole requires {ENV_KMS_VAULT_APPROLE_SECRET_ID} or {ENV_KMS_VAULT_APPROLE_SECRET_ID_FILE} to be set"
)));
}
Ok(VaultAuthMethod::AppRole {
role_id,
secret_id,
secret_id_file,
mount: get_env_str(ENV_KMS_VAULT_APPROLE_MOUNT, DEFAULT_VAULT_APPROLE_MOUNT),
refresh_safety_window_secs: None,
})
}
fn validate_vault_auth_method(backend_name: &str, auth_method: &VaultAuthMethod) -> Result<()> {
match auth_method {
VaultAuthMethod::Token { .. } => Ok(()),
VaultAuthMethod::AppRole {
role_id,
secret_id,
secret_id_file,
mount,
..
} => {
if role_id.is_empty() {
return Err(KmsError::configuration_error(format!("{backend_name} AppRole role_id cannot be empty")));
}
if secret_id.is_empty() && secret_id_file.is_none() {
return Err(KmsError::configuration_error(format!(
"{backend_name} AppRole requires a secret_id or a secret_id_file"
)));
}
if mount.is_empty() {
return Err(KmsError::configuration_error(format!("{backend_name} AppRole mount cannot be empty")));
}
Ok(())
}
VaultAuthMethod::TokenFile {
path,
poll_interval_secs,
..
} => {
if path.as_os_str().is_empty() {
return Err(KmsError::configuration_error(format!("{backend_name} token file path cannot be empty")));
}
if poll_interval_secs == &Some(0) {
return Err(KmsError::configuration_error(format!(
"{backend_name} token file poll interval must be greater than 0"
)));
}
Ok(())
}
}
}
fn validate_vault_development_defaults(
backend_name: &str,
address: &str,
@@ -1106,31 +855,6 @@ mod tests {
assert_eq!(local_config.key_dir, temp_dir.path());
}
#[test]
fn test_oversized_timeout_and_retries_clamped_not_rejected() {
let temp_dir = TempDir::new().expect("Failed to create temp dir");
let config = KmsConfig {
timeout: Duration::from_secs(3_600),
retry_attempts: 50,
..KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults()
};
// Out-of-range values must not keep the service from starting.
assert!(config.validate().is_ok());
assert_eq!(config.effective_timeout(), MAX_OPERATION_TIMEOUT);
assert_eq!(config.effective_retry_attempts(), MAX_RETRY_ATTEMPTS);
// In-range values pass through unchanged.
let config = KmsConfig {
timeout: Duration::from_secs(45),
retry_attempts: 5,
..config
};
assert!(config.validate().is_ok());
assert_eq!(config.effective_timeout(), Duration::from_secs(45));
assert_eq!(config.effective_retry_attempts(), 5);
}
#[test]
fn test_local_development_defaults_require_opt_in() {
let temp_dir = TempDir::new().expect("Failed to create temp dir");
@@ -1253,73 +977,6 @@ mod tests {
assert!(config.vault_config().is_some());
}
#[test]
fn test_persisted_vault_kv2_config_without_mount_path_deserializes() {
// Configurations persisted after mount_path was deprecated may omit the field;
// it must default instead of failing deserialization.
let raw = r#"{
"backend": "VaultKV2",
"backend_config": {
"VaultKV2": {
"address": "http://127.0.0.1:8200",
"auth_method": { "Token": { "token": "t" } },
"namespace": null,
"kv_mount": "secret",
"key_path_prefix": "rustfs/kms/keys",
"tls": null
}
},
"default_key_id": null,
"timeout": {"secs": 30, "nanos": 0},
"retry_attempts": 3,
"enable_cache": true,
"cache_config": {
"max_keys": 1000,
"ttl": {"secs": 3600, "nanos": 0},
"enable_metrics": true
}
}"#;
let config: KmsConfig = serde_json::from_str(raw).expect("persisted kms config without mount_path");
assert_eq!(config.backend, KmsBackend::VaultKv2);
let vault = config.vault_config().expect("vault-kv2 config");
assert_eq!(vault.mount_path, "transit");
}
#[test]
fn test_vault_kv2_empty_mount_path_passes_validation() {
let address = Url::parse("https://vault.example.com:8200").expect("Valid URL");
let mut config = KmsConfig::vault(address, "test-token".to_string());
if let BackendConfig::VaultKv2(vault) = &mut config.backend_config {
vault.mount_path = String::new();
}
assert!(config.validate().is_ok(), "deprecated mount_path must not be required");
}
#[test]
fn test_vault_kv2_sources_do_not_claim_transit_wrapping() {
let sources = [
("config.rs", include_str!("config.rs")),
("api_types.rs", include_str!("api_types.rs")),
("backends/vault.rs", include_str!("backends/vault.rs")),
("lib.rs", include_str!("lib.rs")),
];
// Assemble the needles at runtime so this guard does not match its own source.
let needles = [
format!("wrapping via {}", "Transit"),
format!("KV v2 + {}", "Transit"),
format!("KV2+{}", "Transit"),
format!("you would use Vault's {} engine", "transit"),
];
for (name, source) in sources {
for needle in &needles {
assert!(
!source.contains(needle.as_str()),
"{name} still describes the Vault KV2 backend with `{needle}`"
);
}
}
}
#[test]
fn test_legacy_persisted_vault_transit_config_uses_metadata_defaults() {
let raw = r#"{
@@ -1476,220 +1133,6 @@ mod tests {
);
}
#[test]
fn test_from_env_selects_approle_when_role_id_is_set() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault")),
("RUSTFS_KMS_VAULT_ADDRESS", Some("https://vault.example.com")),
(ENV_KMS_VAULT_APPROLE_ROLE_ID, Some("env-role-id")),
(ENV_KMS_VAULT_APPROLE_SECRET_ID, Some("env-approle-secret-id")),
(ENV_KMS_VAULT_APPROLE_MOUNT, Some("approle-alt")),
// A stale token env var must not override the AppRole selection.
("RUSTFS_KMS_VAULT_TOKEN", Some("vault-token")),
],
|| {
let config = KmsConfig::from_env().expect("kms config should load from env");
let vault = config.vault_config().expect("vault backend config");
let VaultAuthMethod::AppRole {
role_id,
secret_id,
secret_id_file,
mount,
refresh_safety_window_secs,
} = &vault.auth_method
else {
panic!("role id in the environment must select AppRole auth, got {:?}", vault.auth_method);
};
assert_eq!(role_id, "env-role-id");
assert_eq!(secret_id, "env-approle-secret-id");
assert_eq!(secret_id_file, &None);
assert_eq!(mount, "approle-alt");
assert_eq!(refresh_safety_window_secs, &None);
},
);
}
#[test]
fn test_from_env_approle_secret_id_file_is_stored_as_path() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault-transit")),
("RUSTFS_KMS_VAULT_ADDRESS", Some("https://vault.example.com")),
(ENV_KMS_VAULT_APPROLE_ROLE_ID, Some("env-role-id")),
(ENV_KMS_VAULT_APPROLE_SECRET_ID_FILE, Some("/etc/rustfs/approle-secret-id")),
],
|| {
let config = KmsConfig::from_env().expect("kms config should load from env");
let vault = config.vault_transit_config().expect("vault transit backend config");
let VaultAuthMethod::AppRole {
secret_id,
secret_id_file,
mount,
..
} = &vault.auth_method
else {
panic!("role id in the environment must select AppRole auth");
};
// The path is stored, not read: the secret_id file is re-read on
// every login so external rotation is picked up.
assert_eq!(secret_id_file.as_deref(), Some(std::path::Path::new("/etc/rustfs/approle-secret-id")));
assert!(secret_id.is_empty());
assert_eq!(mount, DEFAULT_VAULT_APPROLE_MOUNT);
},
);
}
#[test]
fn test_from_env_approle_requires_secret_id_or_file() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault")),
(ENV_KMS_VAULT_APPROLE_ROLE_ID, Some("env-role-id")),
(ENV_KMS_VAULT_APPROLE_SECRET_ID, None::<&str>),
(ENV_KMS_VAULT_APPROLE_SECRET_ID_FILE, None::<&str>),
],
|| {
let error = KmsConfig::from_env().expect_err("approle without a secret_id source must be rejected");
assert!(error.to_string().contains(ENV_KMS_VAULT_APPROLE_SECRET_ID));
},
);
}
#[test]
fn test_from_env_selects_token_file() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault")),
("RUSTFS_KMS_VAULT_ADDRESS", Some("https://vault.example.com")),
(ENV_KMS_VAULT_TOKEN_FILE, Some("/run/vault-agent/token")),
],
|| {
let config = KmsConfig::from_env().expect("kms config should load from env");
let vault = config.vault_config().expect("vault backend config");
let VaultAuthMethod::TokenFile {
path,
poll_interval_secs,
refresh_safety_window_secs,
} = &vault.auth_method
else {
panic!("token file in the environment must select TokenFile auth, got {:?}", vault.auth_method);
};
assert_eq!(path, std::path::Path::new("/run/vault-agent/token"));
assert_eq!(poll_interval_secs, &None);
assert_eq!(refresh_safety_window_secs, &None);
},
);
}
#[test]
fn test_from_env_token_file_is_mutually_exclusive_with_other_auth() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault")),
(ENV_KMS_VAULT_TOKEN_FILE, Some("/run/vault-agent/token")),
(ENV_KMS_VAULT_APPROLE_ROLE_ID, Some("env-role-id")),
],
|| {
let error = KmsConfig::from_env().expect_err("token file combined with approle must be rejected");
assert!(error.to_string().contains(ENV_KMS_VAULT_TOKEN_FILE));
assert!(error.to_string().contains(ENV_KMS_VAULT_APPROLE_ROLE_ID));
},
);
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault-transit")),
(ENV_KMS_VAULT_TOKEN_FILE, Some("/run/vault-agent/token")),
("RUSTFS_KMS_VAULT_TOKEN", Some("vault-token")),
],
|| {
let error = KmsConfig::from_env().expect_err("token file combined with a static token must be rejected");
assert!(error.to_string().contains(ENV_KMS_VAULT_TOKEN_FILE));
assert!(error.to_string().contains("RUSTFS_KMS_VAULT_TOKEN"));
},
);
}
#[test]
fn test_validate_rejects_bad_token_file_settings() {
let vault_config = |auth_method: VaultAuthMethod| KmsConfig {
backend: KmsBackend::VaultKv2,
backend_config: BackendConfig::VaultKv2(Box::new(VaultConfig {
address: "https://vault.example.com:8200".to_string(),
auth_method,
..Default::default()
})),
..Default::default()
};
let error = vault_config(VaultAuthMethod::token_file(PathBuf::new()))
.validate()
.expect_err("empty token file path must be rejected");
assert!(error.to_string().contains("path"));
let error = vault_config(VaultAuthMethod::TokenFile {
path: PathBuf::from("/run/vault-agent/token"),
poll_interval_secs: Some(0),
refresh_safety_window_secs: None,
})
.validate()
.expect_err("zero poll interval must be rejected");
assert!(error.to_string().contains("poll interval"));
vault_config(VaultAuthMethod::token_file(PathBuf::from("/run/vault-agent/token")))
.validate()
.expect("well-formed token file auth must validate");
}
#[test]
fn test_approle_config_deserializes_legacy_shape_with_defaults() {
// Persisted configurations from before the AppRole implementation only
// carry role_id and secret_id; the new fields must fill with defaults.
let legacy = serde_json::json!({
"AppRole": {
"role_id": "legacy-role",
"secret_id": "legacy-secret-id",
}
});
let auth: VaultAuthMethod = serde_json::from_value(legacy).expect("legacy AppRole config must keep deserializing");
let VaultAuthMethod::AppRole {
role_id,
secret_id_file,
mount,
refresh_safety_window_secs,
..
} = auth
else {
panic!("expected AppRole");
};
assert_eq!(role_id, "legacy-role");
assert_eq!(secret_id_file, None);
assert_eq!(mount, DEFAULT_VAULT_APPROLE_MOUNT);
assert_eq!(refresh_safety_window_secs, None);
}
#[test]
fn test_validate_rejects_incomplete_approle() {
let mut config = KmsConfig::vault_approle(
Url::parse("https://vault.example.com:8200").expect("vault URL"),
String::new(),
"secret-id".to_string(),
);
let error = config.validate().expect_err("empty role_id must be rejected");
assert!(error.to_string().contains("role_id"));
config = KmsConfig::vault_approle(
Url::parse("https://vault.example.com:8200").expect("vault URL"),
"role-id".to_string(),
String::new(),
);
let error = config
.validate()
.expect_err("approle without secret_id or secret_id_file must be rejected");
assert!(error.to_string().contains("secret_id"));
}
#[test]
fn test_from_env_requires_vault_development_opt_in() {
with_vars(
-398
View File
@@ -1,398 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Background worker that completes scheduled key deletions.
//!
//! Every sweep lists keys, picks the ones whose persisted deletion deadline
//! has passed (plus tombstones left by a crashed removal) and hands each to
//! [`KmsBackend::remove_expired_key`], which re-checks state under the
//! backend's own synchronization. The sweep is idempotent and keeps no state
//! of its own, so it is safe to re-run after a restart and safe to run on
//! every node of a deployment concurrently — a key is only ever removed while
//! its (re-read) record is an expired pending deletion or a tombstone.
use crate::backends::{ExpiredKeyRemoval, KmsBackend};
use crate::types::{KeyStatus, ListKeysRequest};
use async_trait::async_trait;
use jiff::Zoned;
use std::sync::Arc;
use std::time::Duration;
use tokio_util::sync::CancellationToken;
use tracing::{debug, info, warn};
/// How often the worker looks for expired pending deletions.
pub const DEFAULT_SWEEP_INTERVAL: Duration = Duration::from_secs(60);
/// Reports configuration that still references a KMS key.
///
/// Consulted before any material is destroyed; a non-empty result blocks the
/// removal until the references disappear. Implementations live where the
/// referencing configuration lives (for example bucket encryption settings in
/// the server) and are injected via
/// [`crate::service_manager::KmsServiceManager::set_deletion_reference_checker`].
#[async_trait]
pub trait DeletionReferenceChecker: Send + Sync {
/// Identifiers of configuration still referencing `key_id` (bucket names,
/// settings paths, ...). Errors must be reported as a reference so that
/// an unavailable checker never unblocks a deletion.
async fn references(&self, key_id: &str) -> Vec<String>;
}
/// Outcome of one sweep, for logging and tests.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct SweepReport {
/// Keys whose record and material were removed this sweep.
pub removed: Vec<String>,
/// Keys left in place because configuration still references them.
pub blocked: Vec<String>,
/// Keys that were pending but not yet due, without a persisted deadline,
/// or whose state changed between inspection and removal.
pub skipped: usize,
/// Keys whose removal attempt failed; retried on the next sweep.
pub failed: usize,
}
pub(crate) struct DeletionWorker {
backend: Arc<dyn KmsBackend>,
default_key_id: Option<String>,
reference_checker: Option<Arc<dyn DeletionReferenceChecker>>,
interval: Duration,
}
impl DeletionWorker {
pub(crate) fn new(
backend: Arc<dyn KmsBackend>,
default_key_id: Option<String>,
reference_checker: Option<Arc<dyn DeletionReferenceChecker>>,
) -> Self {
Self {
backend,
default_key_id,
reference_checker,
interval: DEFAULT_SWEEP_INTERVAL,
}
}
pub(crate) fn spawn(self, cancel: CancellationToken) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move { self.run(cancel).await })
}
async fn run(self, cancel: CancellationToken) {
let mut ticker = tokio::time::interval(self.interval);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
tokio::select! {
_ = cancel.cancelled() => {
debug!("KMS deletion worker stopped");
return;
}
_ = ticker.tick() => {}
}
let report = self.sweep(&Zoned::now()).await;
if !report.removed.is_empty() || !report.blocked.is_empty() || report.failed > 0 {
info!(
removed = ?report.removed,
blocked = ?report.blocked,
skipped = report.skipped,
failed = report.failed,
"KMS deletion sweep completed"
);
}
}
}
/// Run one sweep at the given time. Exposed separately so tests can drive
/// the expiry logic deterministically.
pub(crate) async fn sweep(&self, now: &Zoned) -> SweepReport {
let mut report = SweepReport::default();
let mut marker: Option<String> = None;
loop {
let request = ListKeysRequest {
limit: Some(100),
marker: marker.clone(),
usage_filter: None,
status_filter: None,
};
let response = match self.backend.list_keys(request).await {
Ok(response) => response,
Err(error) => {
warn!(%error, "KMS deletion sweep could not list keys");
report.failed += 1;
return report;
}
};
for key in &response.keys {
if matches!(key.status, KeyStatus::PendingDeletion | KeyStatus::Deleted) {
self.process_key(&key.key_id, now, &mut report).await;
}
}
if !response.truncated {
break;
}
match response.next_marker {
Some(next_marker) => marker = Some(next_marker),
None => break,
}
}
report
}
async fn process_key(&self, key_id: &str, now: &Zoned, report: &mut SweepReport) {
// Never remove a key that live configuration still points at. The
// default key check is built in; broader references (bucket
// encryption settings, ...) come from the injected checker.
if self.default_key_id.as_deref() == Some(key_id) {
warn!(key_id, "expired KMS key is still the default key; refusing removal");
report.blocked.push(key_id.to_string());
return;
}
if let Some(checker) = &self.reference_checker {
let references = checker.references(key_id).await;
if !references.is_empty() {
warn!(key_id, ?references, "expired KMS key is still referenced; refusing removal");
report.blocked.push(key_id.to_string());
return;
}
}
// The backend re-checks state and deadline under its own write
// synchronization, so a cancellation racing this sweep wins there.
match self.backend.remove_expired_key(key_id, now).await {
Ok(ExpiredKeyRemoval::Removed) => report.removed.push(key_id.to_string()),
Ok(ExpiredKeyRemoval::StateChanged | ExpiredKeyRemoval::NotExpired) => report.skipped += 1,
Err(error) => {
warn!(key_id, %error, "failed to remove expired KMS key; will retry next sweep");
report.failed += 1;
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::backends::local::LocalKmsBackend;
use crate::config::KmsConfig;
use crate::error::KmsError;
use crate::types::{CreateKeyRequest, DeleteKeyRequest, DescribeKeyRequest, KeyState, KeyUsage};
async fn local_backend(temp_dir: &tempfile::TempDir) -> Arc<LocalKmsBackend> {
let config = KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults();
Arc::new(LocalKmsBackend::new(config).await.expect("local backend should build"))
}
async fn create_key(backend: &LocalKmsBackend, key_name: &str) -> String {
backend
.create_key(CreateKeyRequest {
key_name: Some(key_name.to_string()),
key_usage: KeyUsage::EncryptDecrypt,
..Default::default()
})
.await
.expect("key should be created")
.key_id
}
async fn schedule(backend: &LocalKmsBackend, key_id: &str) {
backend
.delete_key(DeleteKeyRequest {
key_id: key_id.to_string(),
pending_window_in_days: Some(7),
force_immediate: None,
})
.await
.expect("deletion should be scheduled");
}
fn worker(backend: Arc<LocalKmsBackend>) -> DeletionWorker {
DeletionWorker::new(backend, None, None)
}
fn after_window() -> Zoned {
Zoned::now() + Duration::from_secs(8 * 86400)
}
async fn assert_key_gone(backend: &LocalKmsBackend, key_id: &str) {
let error = backend
.describe_key(DescribeKeyRequest {
key_id: key_id.to_string(),
})
.await
.expect_err("removed key must not be describable");
assert!(matches!(error, KmsError::KeyNotFound { .. }), "expected KeyNotFound, got {error:?}");
}
#[tokio::test]
async fn sweep_removes_expired_pending_key_and_is_idempotent() {
let temp_dir = tempfile::tempdir().expect("temp dir");
let backend = local_backend(&temp_dir).await;
let key_id = create_key(&backend, "expired-key").await;
schedule(&backend, &key_id).await;
let worker = worker(backend.clone());
// Not yet due: nothing happens.
let report = worker.sweep(&Zoned::now()).await;
assert!(report.removed.is_empty());
assert_eq!(report.skipped, 1);
assert_eq!(report.failed, 0);
// Past the deadline: the key is removed.
let report = worker.sweep(&after_window()).await;
assert_eq!(report.removed, vec![key_id.clone()]);
assert_eq!(report.failed, 0);
assert_key_gone(&backend, &key_id).await;
// Re-running the sweep after the removal is a no-op.
let report = worker.sweep(&after_window()).await;
assert_eq!(report, SweepReport::default());
}
#[tokio::test]
async fn cancelled_deletion_always_beats_the_sweep() {
let temp_dir = tempfile::tempdir().expect("temp dir");
let backend = local_backend(&temp_dir).await;
let cancelled = create_key(&backend, "cancelled-key").await;
let doomed = create_key(&backend, "doomed-key").await;
schedule(&backend, &cancelled).await;
schedule(&backend, &doomed).await;
backend
.cancel_key_deletion(crate::types::CancelKeyDeletionRequest {
key_id: cancelled.clone(),
})
.await
.expect("cancel should succeed");
let report = worker(backend.clone()).sweep(&after_window()).await;
assert_eq!(report.removed, vec![doomed.clone()]);
assert_eq!(report.failed, 0);
// The cancelled key survives, enabled and usable.
let described = backend
.describe_key(DescribeKeyRequest {
key_id: cancelled.clone(),
})
.await
.expect("cancelled key must still exist");
assert_eq!(described.key_metadata.key_state, KeyState::Enabled);
assert_key_gone(&backend, &doomed).await;
}
#[tokio::test]
async fn default_key_and_external_references_block_removal() {
struct StaticReferences(Vec<String>);
#[async_trait]
impl DeletionReferenceChecker for StaticReferences {
async fn references(&self, _key_id: &str) -> Vec<String> {
self.0.clone()
}
}
let temp_dir = tempfile::tempdir().expect("temp dir");
let backend = local_backend(&temp_dir).await;
let key_id = create_key(&backend, "referenced-key").await;
schedule(&backend, &key_id).await;
// Blocked while it is the configured default key.
let as_default = DeletionWorker::new(backend.clone(), Some(key_id.clone()), None);
let report = as_default.sweep(&after_window()).await;
assert_eq!(report.blocked, vec![key_id.clone()]);
assert!(report.removed.is_empty());
// Blocked while external configuration still references it.
let with_references = DeletionWorker::new(
backend.clone(),
None,
Some(Arc::new(StaticReferences(vec!["bucket:sse-bucket".to_string()]))),
);
let report = with_references.sweep(&after_window()).await;
assert_eq!(report.blocked, vec![key_id.clone()]);
assert!(report.removed.is_empty());
backend
.describe_key(DescribeKeyRequest { key_id: key_id.clone() })
.await
.expect("blocked key must still exist");
// Removed once nothing references it anymore.
let unreferenced = DeletionWorker::new(backend.clone(), None, Some(Arc::new(StaticReferences(Vec::new()))));
let report = unreferenced.sweep(&after_window()).await;
assert_eq!(report.removed, vec![key_id.clone()]);
}
#[tokio::test]
async fn deadline_survives_backend_restart_and_sweep_completes_it() {
let temp_dir = tempfile::tempdir().expect("temp dir");
let key_id;
{
let backend = local_backend(&temp_dir).await;
key_id = create_key(&backend, "restart-key").await;
schedule(&backend, &key_id).await;
}
// "Restart": a fresh backend over the same directory must still see
// the persisted deadline...
let backend = local_backend(&temp_dir).await;
let described = backend
.describe_key(DescribeKeyRequest { key_id: key_id.clone() })
.await
.expect("key must survive the restart");
assert_eq!(described.key_metadata.key_state, KeyState::PendingDeletion);
assert!(
described.key_metadata.deletion_date.is_some(),
"deletion deadline must survive a backend restart"
);
// ...and the worker completes the deletion without any new schedule call.
let report = worker(backend.clone()).sweep(&after_window()).await;
assert_eq!(report.removed, vec![key_id.clone()]);
assert_key_gone(&backend, &key_id).await;
}
#[tokio::test(start_paused = true)]
async fn worker_loop_removes_due_keys_and_stops_on_cancel() {
let temp_dir = tempfile::tempdir().expect("temp dir");
let backend = local_backend(&temp_dir).await;
let key_id = create_key(&backend, "loop-key").await;
// A zero-day window through the lifecycle client produces a deadline
// that is already due for the worker's wall-clock sweep.
backend
.lifecycle_client()
.schedule_key_deletion(&key_id, 0, None)
.await
.expect("schedule with zero window");
let cancel = CancellationToken::new();
let task = worker(backend.clone()).spawn(cancel.clone());
// The paused clock auto-advances through the worker's interval ticks.
let mut removed = false;
for _ in 0..100 {
tokio::time::sleep(Duration::from_secs(1)).await;
if backend
.describe_key(DescribeKeyRequest { key_id: key_id.clone() })
.await
.is_err()
{
removed = true;
break;
}
}
assert!(removed, "worker loop must remove the due key");
cancel.cancel();
task.await.expect("worker task must stop after cancellation");
}
}
+1 -68
View File
@@ -32,10 +32,7 @@ use std::collections::HashMap;
///
/// This structure stores the encrypted DEK along with metadata needed for decryption.
/// The `master_key_version` field records which version of the KEK (Key Encryption Key)
/// wrapped this DEK so rotation-aware backends can load the matching historical
/// material. Envelopes written before versioning carry `None`; backends must resolve
/// `None` to a deterministic baseline version recorded in key metadata, never
/// implicitly to whatever version is current.
/// was used to encrypt this DEK, enabling proper key rotation support.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DataKeyEnvelope {
pub key_id: String,
@@ -46,12 +43,6 @@ pub struct DataKeyEnvelope {
pub encryption_context: HashMap<String, String>,
#[serde(with = "crate::time_serde::zoned")]
pub created_at: Zoned,
/// KEK version that wrapped `encrypted_key`; `None` on pre-versioning envelopes.
///
/// Optional and omitted when `None` so envelopes from non-rotating backends stay
/// byte-identical to the historical seven-field JSON shape.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub master_key_version: Option<u32>,
}
#[derive(Deserialize)]
@@ -316,7 +307,6 @@ mod tests {
map
},
created_at: Zoned::now(),
master_key_version: None,
};
// Test serialization
@@ -346,8 +336,6 @@ mod tests {
let deserialized: DataKeyEnvelope = serde_json::from_str(envelope_json).expect("Should deserialize current format");
assert_eq!(deserialized.key_id, "test-key-id");
assert_eq!(deserialized.master_key_id, "master-key-id");
// Envelopes persisted before versioning must parse with no master key version.
assert_eq!(deserialized.master_key_version, None);
}
#[tokio::test]
@@ -365,50 +353,6 @@ mod tests {
let deserialized: DataKeyEnvelope = serde_json::from_str(envelope_json).expect("Should deserialize legacy format");
assert_eq!(deserialized.key_id, "test-key-id");
assert_eq!(deserialized.master_key_id, "master-key-id");
assert_eq!(deserialized.master_key_version, None);
}
#[test]
fn test_data_key_envelope_none_version_serializes_without_field() {
// A `None` version must keep the serialized envelope on the historical
// seven-field JSON shape so non-rotating backends emit byte-compatible
// envelopes that older readers accept unchanged.
let envelope = DataKeyEnvelope {
key_id: "test-key-id".to_string(),
master_key_id: "master-key-id".to_string(),
key_spec: "AES_256".to_string(),
encrypted_key: vec![1, 2, 3, 4],
nonce: vec![5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
encryption_context: HashMap::new(),
created_at: Zoned::now(),
master_key_version: None,
};
let value = serde_json::to_value(&envelope).expect("serialize envelope");
let object = value.as_object().expect("envelope serializes to an object");
assert!(!object.contains_key("master_key_version"));
assert_eq!(object.len(), 7, "None version must not change the seven-field JSON shape");
}
#[test]
fn test_data_key_envelope_version_round_trip() {
let envelope = DataKeyEnvelope {
key_id: "test-key-id".to_string(),
master_key_id: "master-key-id".to_string(),
key_spec: "AES_256".to_string(),
encrypted_key: vec![1, 2, 3, 4],
nonce: vec![5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
encryption_context: HashMap::new(),
created_at: Zoned::now(),
master_key_version: Some(7),
};
let serialized = serde_json::to_vec(&envelope).expect("serialize envelope");
let value: serde_json::Value = serde_json::from_slice(&serialized).expect("parse serialized envelope");
assert_eq!(value.get("master_key_version"), Some(&serde_json::json!(7)));
let deserialized: DataKeyEnvelope = serde_json::from_slice(&serialized).expect("deserialize envelope");
assert_eq!(deserialized.master_key_version, Some(7));
}
#[test]
@@ -424,19 +368,8 @@ mod tests {
}"#;
let minio_legacy = br#"{"aead":"AES-256-GCM-HMAC-SHA-256","iv":[1],"nonce":[2],"bytes":[3]}"#;
let duplicate_key_id = [b"{\"key_id\":\"duplicate\",".as_slice(), &kms_envelope[1..]].concat();
// Rotation-aware envelope: the optional master_key_version field must not
// change how mixed batches of old and new envelopes are routed.
let versioned_envelope = {
let mut value: serde_json::Value = serde_json::from_slice(kms_envelope).expect("parse KMS envelope fixture");
value
.as_object_mut()
.expect("KMS envelope fixture is an object")
.insert("master_key_version".to_string(), serde_json::json!(2));
serde_json::to_vec(&value).expect("serialize versioned envelope")
};
assert!(is_data_key_envelope(kms_envelope));
assert!(is_data_key_envelope(&versioned_envelope));
assert!(is_data_key_envelope(&[b" \n".as_slice(), kms_envelope].concat()));
assert!(!is_data_key_envelope(&duplicate_key_id));
assert!(!is_data_key_envelope(b"bm9uY2U=:Y2lwaGVydGV4dA=="));
-104
View File
@@ -90,53 +90,6 @@ pub enum KmsError {
/// Encryption context mismatch
#[error("Encryption context mismatch: {message}")]
ContextMismatch { message: String },
/// Backend operation exceeded its per-attempt timeout or total deadline
#[error("Operation timed out: {message}")]
OperationTimedOut { message: String },
/// Backend operation aborted by cancellation or shutdown
#[error("Operation cancelled: {message}")]
OperationCancelled { message: String },
// New variants must be appended below (never inserted above) so that
// concurrent additions rebase without conflicts.
/// Persisted key material is absent from an otherwise readable key record
#[error(
"Key material missing for key {key_id}: the stored record has no key material; restore it from backup or repair the key explicitly"
)]
MaterialMissing { key_id: String },
/// Persisted key material exists but cannot be decoded
#[error("Key material corrupt for key {key_id}: {message}")]
MaterialCorrupt { key_id: String, message: String },
/// Persisted key material failed authenticated decryption
#[error(
"Key material authentication failed for key {key_id}: the stored material cannot be decrypted with the configured master key"
)]
MaterialAuthenticationFailed { key_id: String },
/// Persisted key record uses a format version unknown to this build
#[error("Unsupported key format version {version:?} for key {key_id}")]
UnsupportedFormatVersion { key_id: String, version: String },
/// Requested master key version has no persisted material for the key
#[error("Key version {version} not found for key {key_id}")]
KeyVersionNotFound { key_id: String, version: u32 },
/// Backup/restore bundle contract violation; see [`crate::backup::BackupError`]
#[error(transparent)]
Backup(#[from] crate::backup::BackupError),
/// Operation is not supported by the active KMS backend
#[error("Operation '{operation}' is not supported by KMS backend '{backend}'")]
UnsupportedCapability { backend: String, operation: String },
/// Backend credentials expired or could not be refreshed in time; requests
/// fail closed instead of being sent with credentials that may lapse mid-flight
#[error("KMS credentials unavailable: {message}")]
CredentialsUnavailable { message: String },
}
impl KmsError {
@@ -234,63 +187,6 @@ impl KmsError {
pub fn context_mismatch<S: Into<String>>(message: S) -> Self {
Self::ContextMismatch { message: message.into() }
}
/// Create an operation timed out error
pub fn operation_timed_out<S: Into<String>>(message: S) -> Self {
Self::OperationTimedOut { message: message.into() }
}
/// Create an operation cancelled error
pub fn operation_cancelled<S: Into<String>>(message: S) -> Self {
Self::OperationCancelled { message: message.into() }
}
/// Create a material missing error
pub fn material_missing<S: Into<String>>(key_id: S) -> Self {
Self::MaterialMissing { key_id: key_id.into() }
}
/// Create a material corrupt error
pub fn material_corrupt<S1: Into<String>, S2: Into<String>>(key_id: S1, message: S2) -> Self {
Self::MaterialCorrupt {
key_id: key_id.into(),
message: message.into(),
}
}
/// Create a material authentication failed error
pub fn material_authentication_failed<S: Into<String>>(key_id: S) -> Self {
Self::MaterialAuthenticationFailed { key_id: key_id.into() }
}
/// Create an unsupported format version error
pub fn unsupported_format_version<S1: Into<String>, S2: Into<String>>(key_id: S1, version: S2) -> Self {
Self::UnsupportedFormatVersion {
key_id: key_id.into(),
version: version.into(),
}
}
/// Create a key version not found error
pub fn key_version_not_found<S: Into<String>>(key_id: S, version: u32) -> Self {
Self::KeyVersionNotFound {
key_id: key_id.into(),
version,
}
}
/// Create an unsupported capability error
pub fn unsupported_capability<S1: Into<String>, S2: Into<String>>(backend: S1, operation: S2) -> Self {
Self::UnsupportedCapability {
backend: backend.into(),
operation: operation.into(),
}
}
/// Create a credentials unavailable error
pub fn credentials_unavailable<S: Into<String>>(message: S) -> Self {
Self::CredentialsUnavailable { message: message.into() }
}
}
/// Convert from standard library errors
+2 -6
View File
@@ -20,7 +20,7 @@
//!
//! ## Features
//!
//! - **Multiple Backends**: Local file storage, Vault KV2 (plain KV storage), and Vault Transit (optional)
//! - **Multiple Backends**: Local file storage, Vault KV2+Transit, and Vault Transit (optional)
//! - **Object Encryption**: Transparent S3-compatible object encryption
//! - **Streaming Encryption**: Memory-efficient encryption for large files
//! - **Key Management**: Full lifecycle management of encryption keys
@@ -66,14 +66,11 @@
// Core modules
pub mod api_types;
pub mod backends;
pub mod backup;
mod cache;
pub mod config;
pub mod deletion_worker;
mod encryption;
mod error;
pub mod manager;
mod policy;
pub mod service;
pub mod service_manager;
mod time_serde;
@@ -87,13 +84,12 @@ pub use api_types::{
UpdateKeyDescriptionRequest, UpdateKeyDescriptionResponse,
};
pub use config::*;
pub use deletion_worker::DeletionReferenceChecker;
pub use encryption::is_data_key_envelope;
pub use error::{KmsError, KmsUnavailableError, Result};
pub use manager::KmsManager;
pub use service::{DataKey, ObjectEncryptionService};
pub use service_manager::{
KmsServiceManager, KmsServiceStatus, KmsStartOutcome, get_global_encryption_service, get_global_kms_service_manager,
KmsServiceManager, KmsServiceStatus, get_global_encryption_service, get_global_kms_service_manager,
init_global_kms_service_manager,
};
pub use types::*;
-90
View File
@@ -66,19 +66,16 @@ impl KmsManager {
}
/// Encrypt data with a master key
#[hotpath::measure]
pub async fn encrypt(&self, request: EncryptRequest) -> Result<EncryptResponse> {
self.backend.encrypt(request).await
}
/// Decrypt data with a master key
#[hotpath::measure]
pub async fn decrypt(&self, request: DecryptRequest) -> Result<DecryptResponse> {
self.backend.decrypt(request).await
}
/// Generate a data encryption key
#[hotpath::measure]
pub async fn generate_data_key(&self, request: GenerateDataKeyRequest) -> Result<GenerateDataKeyResponse> {
self.backend.generate_data_key(request).await
}
@@ -158,51 +155,10 @@ impl KmsManager {
Ok(response)
}
/// Enable a disabled key
pub async fn enable_key(&self, key_id: &str) -> Result<()> {
self.backend.enable_key(key_id).await?;
self.invalidate_cached_metadata(key_id).await;
Ok(())
}
/// Disable a key; existing data remains decryptable
pub async fn disable_key(&self, key_id: &str) -> Result<()> {
self.backend.disable_key(key_id).await?;
self.invalidate_cached_metadata(key_id).await;
Ok(())
}
/// Rotate a key to a new version
pub async fn rotate_key(&self, key_id: &str) -> Result<()> {
self.backend.rotate_key(key_id).await?;
self.invalidate_cached_metadata(key_id).await;
Ok(())
}
/// Drop cached metadata after a state mutation so the next describe
/// observes backend truth instead of the pre-mutation snapshot.
async fn invalidate_cached_metadata(&self, key_id: &str) {
if self.enable_cache {
let mut cache = self.cache.write().await;
cache.remove_key_metadata(key_id).await;
}
}
/// Perform health check on the KMS backend
pub async fn health_check(&self) -> Result<bool> {
self.backend.health_check().await
}
/// Report the capabilities of the configured backend
pub fn backend_capabilities(&self) -> crate::backends::BackendCapabilities {
self.backend.capabilities()
}
/// Direct handle to the configured backend, bypassing the metadata cache.
/// Used by background maintenance that must observe fresh state.
pub(crate) fn backend(&self) -> Arc<dyn KmsBackend> {
self.backend.clone()
}
}
#[cfg(test)]
@@ -263,52 +219,6 @@ mod tests {
assert!(health);
}
#[tokio::test]
async fn lifecycle_round_trip_invalidates_cached_metadata() {
let temp_dir = tempdir().expect("Failed to create temp dir");
let config = KmsConfig::local(temp_dir.path().to_path_buf()).with_insecure_development_defaults();
let backend = Arc::new(LocalKmsBackend::new(config.clone()).await.expect("Failed to create backend"));
let manager = KmsManager::new(backend, config);
let key_id = manager
.create_key(CreateKeyRequest {
key_name: Some("lifecycle-round-trip".to_string()),
..Default::default()
})
.await
.expect("Failed to create key")
.key_id;
let describe = |key_id: String| {
let manager = manager.clone();
async move {
manager
.describe_key(DescribeKeyRequest { key_id })
.await
.expect("describe should succeed")
.key_metadata
.key_state
}
};
// Warm the metadata cache, then flip states; each describe must see
// the post-mutation state, proving the cache entry was dropped.
assert_eq!(describe(key_id.clone()).await, KeyState::Enabled);
manager.disable_key(&key_id).await.expect("disable should succeed");
assert_eq!(describe(key_id.clone()).await, KeyState::Disabled);
manager.enable_key(&key_id).await.expect("enable should succeed");
assert_eq!(describe(key_id.clone()).await, KeyState::Enabled);
// The local backend does not retain version history, so rotation is
// reported as a capability gap rather than a missing key.
let error = manager.rotate_key(&key_id).await.expect_err("local rotate must be rejected");
assert!(
matches!(error, crate::error::KmsError::UnsupportedCapability { .. }),
"expected UnsupportedCapability, got {error:?}"
);
}
#[tokio::test]
async fn generate_data_key_does_not_reuse_context_bound_ciphertext() {
let temp_dir = tempdir().expect("Failed to create temp dir");
File diff suppressed because it is too large Load Diff
-9
View File
@@ -184,15 +184,6 @@ impl ObjectEncryptionService {
self.kms_manager.health_check().await
}
/// Report the capabilities of the configured backend
///
/// # Returns
/// The capability matrix advertised by the active KMS backend
///
pub fn backend_capabilities(&self) -> crate::backends::BackendCapabilities {
self.kms_manager.backend_capabilities()
}
/// Create a data encryption key for object encryption
///
/// # Arguments
+94 -466
View File
@@ -14,23 +14,19 @@
//! KMS service manager for dynamic configuration and runtime management
use crate::backends::vault_credentials::CredentialTaskHandle;
use crate::backends::{KmsBackend, local::LocalKmsBackend};
use crate::config::{BackendConfig, KmsConfig};
use crate::deletion_worker::{DeletionReferenceChecker, DeletionWorker};
use crate::error::{KmsError, Result};
use crate::manager::KmsManager;
use crate::service::ObjectEncryptionService;
use arc_swap::ArcSwap;
use sha2::{Digest, Sha256};
use std::future::Future;
use std::sync::{
Arc, OnceLock,
atomic::{AtomicU64, Ordering},
};
use subtle::ConstantTimeEq;
use tokio::sync::Mutex;
use tokio_util::sync::CancellationToken;
use tokio::sync::{Mutex, RwLock};
use tracing::{debug, error, info, warn};
const LOG_COMPONENT_KMS: &str = "kms";
@@ -97,13 +93,6 @@ pub enum KmsServiceStatus {
Error(String),
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KmsStartOutcome {
Started,
Restarted,
AlreadyRunning,
}
/// Service version information for zero-downtime reconfiguration
#[derive(Clone)]
struct ServiceVersion {
@@ -113,157 +102,77 @@ struct ServiceVersion {
service: Arc<ObjectEncryptionService>,
/// The KMS manager instance
manager: Arc<KmsManager>,
/// Owner of the backend's credential renewal task, if the backend needs
/// one. Stop shuts it down explicitly; reconfigure recycles it through
/// the handle's cancel-on-drop behavior when the old version is discarded.
credential_task: Option<Arc<CredentialTaskHandle>>,
/// Background deletion worker owned by this service version, if the
/// backend supports deletion scheduling
deletion_worker: Option<Arc<DeletionWorkerHandle>>,
}
impl ServiceVersion {
fn shutdown_deletion_worker(&self) {
if let Some(worker) = &self.deletion_worker {
worker.shutdown();
}
}
}
/// Cancellation handle for one service version's deletion worker.
struct DeletionWorkerHandle {
cancel: CancellationToken,
task: std::sync::Mutex<Option<tokio::task::JoinHandle<()>>>,
}
impl DeletionWorkerHandle {
fn shutdown(&self) {
self.cancel.cancel();
if let Ok(mut task) = self.task.lock() {
// Detach: the task observes the cancelled token on its next poll.
drop(task.take());
}
}
}
impl Drop for DeletionWorkerHandle {
fn drop(&mut self) {
// Safety net for versions that are replaced without an explicit stop.
self.cancel.cancel();
}
}
#[derive(Clone)]
struct RuntimeState {
config: Option<KmsConfig>,
status: KmsServiceStatus,
current_service: Option<ServiceVersion>,
}
/// Dynamic KMS service manager with versioned services for zero-downtime reconfiguration
pub struct KmsServiceManager {
/// Atomically published configuration, status, and current service.
state: ArcSwap<RuntimeState>,
/// Current service version (if running)
/// Uses ArcSwap for atomic, lock-free service switching
/// This allows instant atomic updates without blocking readers
current_service: ArcSwap<Option<ServiceVersion>>,
/// Current configuration
config: Arc<RwLock<Option<KmsConfig>>>,
/// Current status
status: Arc<RwLock<KmsServiceStatus>>,
/// Version counter (monotonically increasing)
version_counter: Arc<AtomicU64>,
/// Mutex to protect lifecycle operations (start, stop, reconfigure)
/// This ensures only one lifecycle operation happens at a time
lifecycle_mutex: Arc<Mutex<()>>,
/// External reference checker consulted before expired keys are removed
deletion_reference_checker: std::sync::RwLock<Option<Arc<dyn DeletionReferenceChecker>>>,
}
impl KmsServiceManager {
/// Create a new KMS service manager (not configured)
pub fn new() -> Self {
Self {
state: ArcSwap::from_pointee(RuntimeState {
config: None,
status: KmsServiceStatus::NotConfigured,
current_service: None,
}),
current_service: ArcSwap::from_pointee(None),
config: Arc::new(RwLock::new(None)),
status: Arc::new(RwLock::new(KmsServiceStatus::NotConfigured)),
version_counter: Arc::new(AtomicU64::new(0)),
lifecycle_mutex: Arc::new(Mutex::new(())),
deletion_reference_checker: std::sync::RwLock::new(None),
}
}
/// Install the reference checker consulted before the deletion worker
/// removes an expired key. Takes effect for workers spawned by the next
/// start or reconfigure.
pub fn set_deletion_reference_checker(&self, checker: Arc<dyn DeletionReferenceChecker>) {
if let Ok(mut slot) = self.deletion_reference_checker.write() {
*slot = Some(checker);
}
}
fn deletion_reference_checker(&self) -> Option<Arc<dyn DeletionReferenceChecker>> {
self.deletion_reference_checker.read().ok().and_then(|slot| slot.clone())
}
/// Get current service status
pub async fn get_status(&self) -> KmsServiceStatus {
self.state.load().status.clone()
self.status.read().await.clone()
}
/// Get current configuration (if any)
pub async fn get_config(&self) -> Option<KmsConfig> {
self.state.load().config.clone()
self.config.read().await.clone()
}
/// Get configuration for status and management responses without static key material.
pub async fn get_redacted_config(&self) -> Option<KmsConfig> {
let mut config = self.state.load().config.clone()?;
Self::redact_config(&mut config);
Some(config)
}
/// Get status and redacted configuration from the same published snapshot.
pub async fn get_redacted_state(&self) -> (KmsServiceStatus, Option<KmsConfig>) {
let state = self.state.load();
let mut config = state.config.clone();
if let Some(config) = &mut config {
Self::redact_config(config);
}
(state.status.clone(), config)
}
fn redact_config(config: &mut KmsConfig) {
let mut config = self.config.read().await.clone()?;
if let BackendConfig::Static(static_config) = &mut config.backend_config {
use zeroize::Zeroize;
static_config.secret_key.zeroize();
}
Some(config)
}
/// Configure KMS with new configuration
pub async fn configure(&self, new_config: KmsConfig) -> Result<()> {
self.configure_with_persistence(new_config, || async { Ok(()) }).await
}
/// Configure KMS and publish the in-memory state only after persistence succeeds.
///
/// The persistence callback runs under the lifecycle lock and must not call
/// another lifecycle method on this manager.
pub async fn configure_with_persistence<Persist, PersistFuture>(&self, new_config: KmsConfig, persist: Persist) -> Result<()>
where
Persist: FnOnce() -> PersistFuture,
PersistFuture: Future<Output = Result<()>>,
{
new_config.validate()?;
let _guard = self.lifecycle_mutex.lock().await;
let current = self.state.load_full();
validate_local_transition(current.config.as_ref(), &new_config)?;
if current.current_service.is_some() {
return Err(KmsError::configuration_error(
"Cannot configure KMS while it is running; use reconfigure instead",
));
new_config.validate()?;
{
let config = self.config.read().await;
validate_local_transition(config.as_ref(), &new_config)?;
}
// Update configuration
{
let mut config = self.config.write().await;
*config = Some(new_config.clone());
}
// Update status
{
let mut status = self.status.write().await;
*status = KmsServiceStatus::Configured;
}
persist().await?;
self.state.store(Arc::new(RuntimeState {
config: Some(new_config),
status: KmsServiceStatus::Configured,
current_service: None,
}));
debug!(
event = EVENT_KMS_SERVICE_STATE,
@@ -281,35 +190,19 @@ impl KmsServiceManager {
self.start_internal().await
}
/// Start or restart KMS with the running-state decision serialized with the lifecycle action.
pub async fn start_or_restart(&self, force: bool) -> Result<KmsStartOutcome> {
let _guard = self.lifecycle_mutex.lock().await;
let running = self.state.load().current_service.is_some();
if running && !force {
return Ok(KmsStartOutcome::AlreadyRunning);
}
self.start_internal().await?;
Ok(if running {
KmsStartOutcome::Restarted
} else {
KmsStartOutcome::Started
})
}
/// Internal start implementation (called within lifecycle mutex)
async fn start_internal(&self) -> Result<()> {
let state = self.state.load_full();
let config = match state.config.as_ref() {
Some(config) => config.clone(),
None => {
let err_msg = "Cannot start KMS: no configuration provided";
error!("{}", err_msg);
self.state.store(Arc::new(RuntimeState {
config: None,
status: KmsServiceStatus::Error(err_msg.to_string()),
current_service: None,
}));
return Err(KmsError::configuration_error(err_msg));
let config = {
let config_guard = self.config.read().await;
match config_guard.as_ref() {
Some(config) => config.clone(),
None => {
let err_msg = "Cannot start KMS: no configuration provided";
error!("{}", err_msg);
let mut status = self.status.write().await;
*status = KmsServiceStatus::Error(err_msg.to_string());
return Err(KmsError::configuration_error(err_msg));
}
}
};
@@ -322,9 +215,17 @@ impl KmsServiceManager {
"KMS service starting"
);
match self.create_healthy_service_version(&config).await {
match self.create_service_version(&config).await {
Ok(service_version) => {
self.publish_running(config, service_version);
// Atomically update to new service version (lock-free, instant)
// ArcSwap::store() is a true atomic operation using CAS
self.current_service.store(Arc::new(Some(service_version)));
// Update status
{
let mut status = self.status.write().await;
*status = KmsServiceStatus::Running;
}
debug!(
event = EVENT_KMS_SERVICE_STATE,
@@ -338,24 +239,13 @@ impl KmsServiceManager {
Err(e) => {
let err_msg = format!("Failed to create KMS backend: {e}");
error!("{}", err_msg);
if state.current_service.is_none() {
self.state.store(Arc::new(RuntimeState {
config: state.config.clone(),
status: KmsServiceStatus::Error(err_msg.clone()),
current_service: None,
}));
}
let mut status = self.status.write().await;
*status = KmsServiceStatus::Error(err_msg.clone());
Err(KmsError::backend_error(&err_msg))
}
}
}
/// Replace the running service without exposing a stopped interval.
pub async fn restart(&self) -> Result<()> {
let _guard = self.lifecycle_mutex.lock().await;
self.start_internal().await
}
/// Stop KMS service
///
/// Note: This stops accepting new operations, but existing operations using
@@ -377,25 +267,14 @@ impl KmsServiceManager {
// Atomically clear current service version (lock-free, instant)
// Note: Existing Arc references will keep the service alive until operations complete
let state = self.state.load_full();
if let Some(current) = state.current_service.as_ref() {
current.shutdown_deletion_worker();
}
self.state.store(Arc::new(RuntimeState {
config: state.config.clone(),
status: if state.config.is_some() {
KmsServiceStatus::Configured
} else {
KmsServiceStatus::NotConfigured
},
current_service: None,
}));
self.current_service.store(Arc::new(None));
// Shut down the stopped version's credential renewal task before
// reporting stopped, so stop deterministically recycles the background
// task even while in-flight operations still hold the old service Arc.
if let Some(task) = state.current_service.as_ref().and_then(|sv| sv.credential_task.clone()) {
task.shutdown().await;
// Update status (keep configuration)
{
let mut status = self.status.write().await;
if !matches!(*status, KmsServiceStatus::NotConfigured) {
*status = KmsServiceStatus::Configured;
}
}
debug!(
@@ -419,22 +298,6 @@ impl KmsServiceManager {
/// This ensures zero downtime during reconfiguration, even for long-running
/// operations like encrypting large files.
pub async fn reconfigure(&self, new_config: KmsConfig) -> Result<()> {
self.reconfigure_with_persistence(new_config, || async { Ok(()) }).await
}
/// Reconfigure KMS after the candidate is healthy and persistence succeeds.
///
/// The persistence callback runs under the lifecycle lock and must not call
/// another lifecycle method on this manager.
pub async fn reconfigure_with_persistence<Persist, PersistFuture>(
&self,
new_config: KmsConfig,
persist: Persist,
) -> Result<()>
where
Persist: FnOnce() -> PersistFuture,
PersistFuture: Future<Output = Result<()>>,
{
let _guard = self.lifecycle_mutex.lock().await;
debug!(
@@ -445,18 +308,33 @@ impl KmsServiceManager {
"KMS service reconfiguring"
);
new_config.validate()?;
validate_local_transition(self.state.load().config.as_ref(), &new_config)?;
{
let config = self.config.read().await;
validate_local_transition(config.as_ref(), &new_config)?;
}
// Create new service version without stopping old one
// This allows existing operations to continue while new operations use new service
match self.create_healthy_service_version(&new_config).await {
match self.create_service_version(&new_config).await {
Ok(new_service_version) => {
// Get old version for logging (lock-free read)
let old_version = self.state.load().current_service.as_ref().map(|sv| sv.version);
let old_version = self.current_service.load().as_ref().as_ref().map(|sv| sv.version);
persist().await?;
{
let mut config = self.config.write().await;
*config = Some(new_config);
}
self.publish_running(new_config, new_service_version.clone());
// Atomically switch to new service version (lock-free, instant CAS operation)
// This is a true atomic operation - no waiting for locks, instant switch
// Old service will be dropped when no more Arc references exist
self.current_service.store(Arc::new(Some(new_service_version.clone())));
// Update status
{
let mut status = self.status.write().await;
*status = KmsServiceStatus::Running;
}
if let Some(old_ver) = old_version {
info!(
@@ -493,7 +371,7 @@ impl KmsServiceManager {
/// Returns the manager from the current service version.
/// Uses lock-free atomic load for optimal performance.
pub async fn get_manager(&self) -> Option<Arc<KmsManager>> {
self.state.load().current_service.as_ref().map(|sv| sv.manager.clone())
self.current_service.load().as_ref().as_ref().map(|sv| sv.manager.clone())
}
/// Get encryption service (if running)
@@ -503,7 +381,7 @@ impl KmsServiceManager {
/// This ensures new operations always use the latest service version,
/// while existing operations continue using their Arc references.
pub async fn get_encryption_service(&self) -> Option<Arc<ObjectEncryptionService>> {
self.state.load().current_service.as_ref().map(|sv| sv.service.clone())
self.current_service.load().as_ref().as_ref().map(|sv| sv.service.clone())
}
/// Get current service version number
@@ -511,16 +389,14 @@ impl KmsServiceManager {
/// Useful for monitoring and debugging.
/// Uses lock-free atomic load.
pub async fn get_service_version(&self) -> Option<u64> {
self.state.load().current_service.as_ref().map(|sv| sv.version)
self.current_service.load().as_ref().as_ref().map(|sv| sv.version)
}
/// Health check for the KMS service
pub async fn health_check(&self) -> Result<bool> {
let checked_state = self.state.load_full();
match checked_state.current_service.as_ref() {
Some(service_version) => {
let manager = service_version.manager.clone();
let checked_version = service_version.version;
let manager = self.get_manager().await;
match manager {
Some(manager) => {
// Perform health check on the backend
match manager.health_check().await {
Ok(healthy) => {
@@ -531,8 +407,9 @@ impl KmsServiceManager {
}
Err(e) => {
error!("KMS health check error: {}", e);
let _guard = self.lifecycle_mutex.lock().await;
self.mark_health_error_if_current(checked_version, &e);
// Update status to error
let mut status = self.status.write().await;
*status = KmsServiceStatus::Error(format!("Health check failed: {e}"));
Err(e)
}
}
@@ -555,10 +432,7 @@ impl KmsServiceManager {
info!("Creating KMS service version {} with backend: {:?}", version, config.backend);
// Create backend. Vault backends may also spawn a background
// credential renewal task whose owner handle lives on the service
// version, so replacing the version recycles the task.
let mut credential_task = None;
// Create backend
let backend = match &config.backend_config {
BackendConfig::Local(_) => {
info!("Creating Local KMS backend for version {}", version);
@@ -568,13 +442,11 @@ impl KmsServiceManager {
BackendConfig::VaultKv2(_) => {
info!("Creating Vault KV2 KMS backend for version {}", version);
let backend = crate::backends::vault::VaultKmsBackend::new(config.clone()).await?;
credential_task = backend.spawn_credential_renewal().map(Arc::new);
Arc::new(backend) as Arc<dyn KmsBackend>
}
BackendConfig::VaultTransit(_) => {
info!("Creating Vault Transit KMS backend for version {}", version);
let backend = crate::backends::vault_transit::VaultTransitKmsBackend::new(config.clone()).await?;
credential_task = backend.spawn_credential_renewal().map(Arc::new);
Arc::new(backend) as Arc<dyn KmsBackend>
}
BackendConfig::Static(_) => {
@@ -594,59 +466,8 @@ impl KmsServiceManager {
version,
service: encryption_service,
manager: kms_manager,
credential_task,
deletion_worker: None,
})
}
async fn create_healthy_service_version(&self, config: &KmsConfig) -> Result<ServiceVersion> {
let service_version = self.create_service_version(config).await?;
if !service_version.manager.health_check().await? {
return Err(KmsError::backend_error("KMS backend health check failed"));
}
Ok(service_version)
}
fn publish_running(&self, config: KmsConfig, mut service_version: ServiceVersion) {
if let Some(previous) = self.state.load().current_service.as_ref() {
previous.shutdown_deletion_worker();
}
service_version.deletion_worker = self.spawn_deletion_worker(&config, &service_version);
self.state.store(Arc::new(RuntimeState {
config: Some(config),
status: KmsServiceStatus::Running,
current_service: Some(service_version),
}));
}
/// Spawn the background deletion worker for a service version about to be
/// published, if its backend supports deletion scheduling. The worker is
/// only started at publish time so failed start/reconfigure candidates
/// never leak a running task.
fn spawn_deletion_worker(&self, config: &KmsConfig, service_version: &ServiceVersion) -> Option<Arc<DeletionWorkerHandle>> {
let backend = service_version.manager.backend();
if !backend.capabilities().schedule_deletion {
return None;
}
let cancel = CancellationToken::new();
let worker = DeletionWorker::new(backend, config.default_key_id.clone(), self.deletion_reference_checker());
let task = worker.spawn(cancel.clone());
Some(Arc::new(DeletionWorkerHandle {
cancel,
task: std::sync::Mutex::new(Some(task)),
}))
}
fn mark_health_error_if_current(&self, checked_version: u64, error: &KmsError) {
let current = self.state.load_full();
if current.current_service.as_ref().map(|version| version.version) == Some(checked_version) {
self.state.store(Arc::new(RuntimeState {
config: current.config.clone(),
status: KmsServiceStatus::Error(format!("Health check failed: {error}")),
current_service: current.current_service.clone(),
}));
}
}
}
impl Default for KmsServiceManager {
@@ -679,11 +500,6 @@ pub async fn get_global_encryption_service() -> Option<Arc<ObjectEncryptionServi
#[cfg(test)]
mod tests {
use super::*;
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
fn static_config(key_id: &str, fill: u8) -> KmsConfig {
KmsConfig::static_kms(key_id.to_string(), BASE64_STANDARD.encode([fill; 32]))
}
#[tokio::test]
async fn configure_rejects_insecure_development_defaults_before_state_update() {
@@ -701,6 +517,8 @@ mod tests {
#[tokio::test]
async fn redacted_config_omits_static_key_material() {
use base64::Engine as _;
let manager = KmsServiceManager::new();
let encoded_key = base64::engine::general_purpose::STANDARD.encode([0x5au8; 32]);
manager
@@ -715,136 +533,6 @@ mod tests {
assert!(static_config.secret_key.is_empty());
}
#[tokio::test]
async fn configure_persistence_failure_leaves_state_unchanged() {
let manager = KmsServiceManager::new();
let result = manager
.configure_with_persistence(static_config("key-a", 0x11), || async { Err(KmsError::backend_error("persist failed")) })
.await;
assert!(result.is_err());
assert_eq!(manager.get_status().await, KmsServiceStatus::NotConfigured);
assert!(manager.get_config().await.is_none());
assert!(manager.get_encryption_service().await.is_none());
}
#[tokio::test]
async fn configure_rejects_running_service_without_changing_snapshot() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
let version = manager.get_service_version().await;
let result = manager.configure(static_config("key-b", 0x22)).await;
assert!(result.is_err());
assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
assert_eq!(manager.get_service_version().await, version);
assert_eq!(
manager.get_config().await.and_then(|config| config.default_key_id),
Some("key-a".to_string())
);
}
#[tokio::test]
async fn reconfigure_persistence_failure_keeps_old_running_snapshot() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
let old_version = manager.get_service_version().await;
let old_service = manager.get_encryption_service().await.expect("old service");
let result = manager
.reconfigure_with_persistence(static_config("key-b", 0x22), || async {
Err(KmsError::backend_error("persist failed"))
})
.await;
assert!(result.is_err());
assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
assert_eq!(manager.get_service_version().await, old_version);
assert_eq!(
manager.get_config().await.and_then(|config| config.default_key_id),
Some("key-a".to_string())
);
assert!(Arc::ptr_eq(
&old_service,
&manager.get_encryption_service().await.expect("old service remains")
));
}
#[tokio::test]
async fn reconfigure_candidate_failure_keeps_old_running_snapshot() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
let old_version = manager.get_service_version().await;
let invalid_parent = tempfile::NamedTempFile::new().expect("temporary file");
let invalid_config = KmsConfig::local(invalid_parent.path().join("keys")).with_insecure_development_defaults();
let result = manager.reconfigure(invalid_config).await;
assert!(result.is_err());
assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
assert_eq!(manager.get_service_version().await, old_version);
assert_eq!(
manager.get_config().await.and_then(|config| config.default_key_id),
Some("key-a".to_string())
);
}
#[tokio::test]
async fn restart_never_unpublishes_the_running_service() {
let manager = Arc::new(KmsServiceManager::new());
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
let old_version = manager.get_service_version().await.expect("old version");
let restarting = {
let manager = manager.clone();
tokio::spawn(async move { manager.restart().await })
};
while !restarting.is_finished() {
assert!(manager.get_encryption_service().await.is_some());
tokio::task::yield_now().await;
}
restarting.await.expect("restart task").expect("restart");
assert!(manager.get_encryption_service().await.is_some());
assert!(manager.get_service_version().await.expect("new version") > old_version);
assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
}
#[tokio::test]
async fn start_or_restart_decides_under_the_lifecycle_lock() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
assert_eq!(manager.start_or_restart(false).await.expect("initial start"), KmsStartOutcome::Started);
let first_version = manager.get_service_version().await.expect("first version");
assert_eq!(
manager.start_or_restart(false).await.expect("already running"),
KmsStartOutcome::AlreadyRunning
);
assert_eq!(manager.get_service_version().await, Some(first_version));
assert_eq!(manager.start_or_restart(true).await.expect("forced restart"), KmsStartOutcome::Restarted);
assert!(manager.get_service_version().await.expect("restarted version") > first_version);
}
#[tokio::test]
async fn stale_health_failure_cannot_poison_new_service_status() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
let old_version = manager.get_service_version().await.expect("old version");
manager.restart().await.expect("restart");
manager.mark_health_error_if_current(old_version, &KmsError::backend_error("stale failure"));
assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
}
#[tokio::test]
async fn forbidden_local_master_key_change_preserves_running_config_and_service() {
use crate::types::{CreateKeyRequest, KeyUsage};
@@ -920,66 +608,6 @@ mod tests {
assert!(matches!(current.backend_config, BackendConfig::Local(_)));
}
#[tokio::test]
async fn deletion_worker_follows_the_service_lifecycle() {
use tempfile::TempDir;
let key_dir = TempDir::new().expect("create local KMS directory");
let mut config = KmsConfig::local(key_dir.path().to_path_buf());
config.allow_insecure_dev_defaults = true;
let manager = KmsServiceManager::new();
manager.configure(config).await.expect("configure local KMS");
manager.start().await.expect("start local KMS");
let first_worker = manager
.state
.load()
.current_service
.as_ref()
.expect("running service")
.deletion_worker
.clone()
.expect("local backend must run a deletion worker");
assert!(!first_worker.cancel.is_cancelled());
// Replacing the service version replaces (and cancels) its worker.
manager.restart().await.expect("restart");
assert!(first_worker.cancel.is_cancelled(), "replaced version's worker must be cancelled");
let second_worker = manager
.state
.load()
.current_service
.as_ref()
.expect("running service")
.deletion_worker
.clone()
.expect("restarted service must run a fresh worker");
assert!(!second_worker.cancel.is_cancelled());
// Stopping the service stops its worker.
manager.stop().await.expect("stop");
assert!(second_worker.cancel.is_cancelled(), "stop must cancel the deletion worker");
}
#[tokio::test]
async fn static_backend_runs_no_deletion_worker() {
let manager = KmsServiceManager::new();
manager.configure(static_config("key-a", 0x11)).await.expect("configure");
manager.start().await.expect("start");
assert!(
manager
.state
.load()
.current_service
.as_ref()
.expect("running service")
.deletion_worker
.is_none(),
"a backend without deletion scheduling must not run a worker"
);
}
#[tokio::test]
async fn reconfigure_allows_safe_local_runtime_settings_only() {
use tempfile::TempDir;

Some files were not shown because too many files have changed in this diff Show More